The black box excuse
The black box excuse has an expiry date. From 2 August 2026, the EU AI Act's high risk obligations are enforceable, and 78% of organisations have not taken meaningful steps towards compliance (Vision Compliance, 2026 EU AI Act Readiness Report).
Plenty of them have a reason ready, and it is the same reason it has always been. You can't fully audit AI, the model is a black box.
What the box is hiding
The excuse has a truth inside it, which is why it has survived this long. Nobody can point at a neural network's weights and explain why it chose what it chose. Explainability is a genuinely hard research problem. We need to look at what the excuse is hiding, it takes the one part of the system that can’t be proven and uses it to avoid proving the five parts that can.
Every AI-enabled system, however inscrutable its model, has a perfectly knowable perimeter. What task went in. Which model, and which version of the rules, was in force at the time. What actions the system took and what data it touched. What it cost. What shipped as a result. None of that needs a research breakthrough, it needs engineering that has existed for decades, and almost nobody has built it. 61% of organisations have no process for producing the technical documentation the Act requires for high risk systems.
We run an agentic development pipeline, and every task in it carries a single correlation identifier from intake to delivery. Every model call, every action, every rule in force at the time, stitched to that one identifier in an immutable audit log. Asking what the AI did, and under what authority, is a query that takes minutes. The model at the centre is as much of a black box as anyone's. It has never mattered less.
Most organisations have not built this because they do not want it. A workflow audit trail would show how little of their AI estate anyone is actually watching, and an excuse that sounds like a physics problem is more comfortable than an answer that looks like negligence.
The law has stopped accepting the excuse
Read what the Act actually demands of high risk systems. Article 12 requires automatic recording of events across the system's lifetime, so that its operation is traceable. Article 11 requires technical documentation of how the system was built and how it is governed. Nowhere does it require you to explain your model's internal weights. The regulation has quietly ruled on the black box excuse, it legislates proof of the provable parts and leaves the research problem to the researchers. And the obligations fall on deployers too, a SaaS business using a third party model for credit scoring or recruitment carries them just as the lab that trained the model does.
For regulated SaaS businesses, the question is which side of that line you are standing on. Scail's AI Risk Value Index measures exactly this, whether an organisation can prove how its AI systems behave or is still gesturing at the box.
What boards need to see now
Boards are now signing off AI risk they cannot currently evidence. Ask for the proof behind your AI estate and most organisations will produce a policy document, which is a statement of intent, and intent is precisely what regulators have stopped accepting.
The Scail AI Risk & Value Scorecard assesses AI capability across eight areas, from governance and risk through to execution and value realisation, and it treats auditability as the thread running through all of them, because a capability you cannot evidence is a liability wearing a nice dashboard.
The winners under the AI Act will not be the businesses with the most explainable models. They will be the ones that can answer, in minutes, the question the regulator is now entitled to ask. What did your AI do, and who said it could?
Read more about our AI Risk & Value Scorecard.
