BlackgradeSystems

Field note AI Act

The AI Act for people actually building AI products

Reading the regulation as a set of system requirements.

Regulation (EU) 2024/1689 is long, and most summaries written for engineers stop at the risk pyramid. The pyramid is the easy part. What matters for a build is the list of artefacts you have to be able to produce, and when they have to exist.

01

Two questions before anything else

The first question is what role you occupy. The obligations of a provider, who develops a system and places it on the market under its own name, are not the obligations of a deployer, who uses it under its own authority. Integrating a third party model into your product and offering it to customers can make you a provider of the resulting system even though you trained nothing.

The second question is what the system does, not what it is built from. Classification follows purpose and context of use. The same language model can sit inside a system that is minimal risk in one product and high risk in another, because the regulation looks at the function performed, not at the architecture.

Teams that answer these two questions late rebuild. Teams that answer them in the design phase usually discover the classification is manageable and the work is bounded.

02

The tiers, stated as consequences

General purpose models carry their own layer of obligations, principally documentation and information to be passed downstream, with additional requirements where a model is considered to present systemic risk. If you build on someone else model, the practical question is whether they will give you what you need to document your own system.

TierWhat it means for a buildNote
Prohibited A short list of practices that may not be placed on the market or used at all. Check first, it is quick
High risk Permitted subject to a full compliance programme: risk management, data governance, technical documentation, logging, transparency, human oversight, accuracy and robustness, conformity assessment. The bulk of the work
Transparency obligations Certain systems must disclose their nature to the people interacting with them, and certain generated content must be marked. Cheap, often forgotten
Minimal risk No specific obligations under the Act, which does not exempt you from everything else that applies. Most systems
03

The artefacts a high risk system has to produce

Read as an engineering backlog rather than as legal text, the high risk regime asks for a small number of things that have to exist and stay current.

Two of these are pure engineering and are usually underestimated. Logging has to be designed, because "we have application logs" is not the same as being able to reconstruct why a specific decision came out the way it did for a specific person on a specific day. And human oversight has to be effective, which means the human needs enough context and enough authority to actually change the outcome. A confirmation button on a screen that shows nothing but a score is oversight in name only.

  • A risk management process that runs across the lifecycle, not a document written once
  • Data governance covering the provenance, relevance and known limitations of training, validation and test data
  • Technical documentation complete enough for an assessor to understand how the system works
  • Automatic logging of events over the system lifetime, sufficient to trace behaviour after the fact
  • Instructions for use that let a deployer operate it correctly
  • Human oversight designed into the system rather than promised in a policy
  • A stated level of accuracy and robustness, with the metrics that support it
  • A quality management system, and a conformity assessment before placing on the market
04

Designing oversight that survives contact with reality

Effective oversight has three properties. The reviewer sees the inputs that drove the output, not only the output. The reviewer can override, and the override is recorded with a reason. And the rate of overrides is monitored, because an oversight step with a zero percent override rate over months is either a perfect model or a rubber stamp, and it is rarely the first.

Building the override path first, before the automation, tends to produce better systems. It forces the question of what the human would need to decide, which is usually the same context the model should have had.

A metric worth having Track override rate and time to decision per reviewer and per cohort. Both collapsing towards zero is the earliest signal that a control has become ceremonial.
05

Timing, and why it does not change the plan

The Act entered into force in 2024 and applies in stages, with prohibitions first, obligations for general purpose models next, and the high risk regime later. The staging matters for legal exposure and barely at all for engineering planning, because the artefacts above take longer to build than the gap between phases.

The practical advice is unglamorous. Classify now, in writing, with the reasoning. Build logging and oversight into the first version. Keep the documentation in the repository next to the code so it is updated by the same change that updates behaviour.

06

The underlying request

Strip the legal structure away and the regulation asks for something engineers already recognise: know what your system is for, know what data it learned from, be able to explain a specific output after the fact, and make sure a person can intervene.

Systems built that way are easier to debug, easier to sell into regulated customers, and easier to defend. The compliance argument is the weakest reason to build them.

Contact

Working on something in this territory?

Financial infrastructure, regulated systems, AI in controlled environments, cryptography, platforms at scale.

Get in touch