The phrase "black box AI" smuggles in an assumption that deserves to be dragged into the light: that opacity is a technical inevitability — the unavoidable price of powerful models. Billions of parameters, inscrutable by nature; what can anyone do?
Spend time around real deployments and you discover the assumption is false in a way that changes the policy conversation entirely. There are actually two kinds of black box in the wild, and only one of them has anything to do with technology.
Box one: opaque by nature. Box two: opaque by decision.
The first black box is the famous one — the genuinely inscrutable model, complexity beyond human tracing. Real, and really hard.
The second is a business practice. Consider loan risk models. Some companies build simple models — the kind that could be printed on a page and understood by any loan officer, or any applicant — and then tell no one what's in them, because the model is proprietary. The technology is perfectly transparent; the institution is not. The applicant experiences a decision that cannot be questioned. From outside, it's indistinguishable from the most inscrutable neural network — and it was a choice.
And the two boxes compound: many organizations deploy complex models knowing simpler ones would perform comparably, and then keep those hidden too. Complexity as moat, wrapped in secrecy as policy — with "the AI is a black box" available as an all-purpose deflection.
Once you see the two boxes, the phrase "we can't explain the model" becomes ambiguous in a way that should make everyone — regulators especially — reach for a follow-up question: can't, or won't? Conflating them lets deliberate opacity free-ride on technical mystique. The inscrutability of frontier models provides rhetorical cover for garden-variety secrecy about systems that could be published on a single page.
Complexity is often a downgrade
The cover story weakens further when you examine where complex models actually outperform. The honest technical answer: far fewer places than deployment patterns suggest. On clean, high-signal data — images, language — complexity pays. But an enormous share of consequential decisions run on noisy tabular data: sparse records of human lives and behavior, where outcomes are genuinely uncertain. Criminal justice data, rife with documented inconsistencies that can affect a person's freedom. Healthcare records, notoriously messy. Financial histories.
On such data, the noise ceiling arrives fast, and piling on complexity past it doesn't extract more truth — it overfits to static. Well-built simple models — decision trees, additive models, scoring systems — routinely match black-box performance there. Which yields a conclusion that still hasn't penetrated procurement: for noisy, high-stakes tabular problems, the complex black box is frequently a strict downgrade — equal accuracy, worse troubleshootability, at the exact moment troubleshootability is everything. With an opaque model, you often can't tell it's wrong until the harm has already happened; a transparent one lets the people using it catch the failure before it lands. Choosing opacity there isn't a trade-off. It's negligence with a technology story.
The root problem is a missing distinction in most organizations: when is a black box necessary, and when is it merely fashionable? That's a learnable distinction — signal-rich perception problems on one side, noisy human-outcome problems on the other — but almost nobody procuring these systems has been taught it. So the default follows fashion, and fashion says bigger.
The disclosure principle
All of which leads to a policy idea I've come to support, and it's refreshingly modest. Not "ban black boxes." Not "mandate explainability everywhere." Just this: people should know when a black box is being used in a high-stakes decision where it isn't needed. Disclosure, at minimum — arguably by legislation, since no market force currently supplies it.
The applicant denied a loan, the defendant scored for risk, the patient triaged by algorithm — each has a reasonable claim to one fact: was this decision made by a system that could have been transparent at no cost to accuracy, but wasn't? Notice the incentive structure that mere sunlight creates. If an institution must disclose "we used an opaque model where a transparent one would have performed equivalently," the sentence indicts itself — so institutions acquire a reason to either justify the opacity or abandon it. The regulation barely needs teeth; the embarrassment is the mechanism. We don't need to open every box by force. We need to make unnecessary boxes indefensible in public.
I'd add the obvious complement: opacity-by-secrecy deserves separate scrutiny from opacity-by-complexity. Trade-secret protection for a genuinely novel architecture is one conversation; trade-secret protection for a ten-variable scoring formula deciding people's housing is another. The second is secrecy about rules governing citizens, and due process has never looked kindly on secret rules.
Where this actually lands
Which future we get won't be decided by technology alone — commercial secrecy, competitive pressure, user expectations, and legislation are all pulling on the outcome, and openness wins only if the demand side insists on it. So, three habits worth adopting now. If you build: before reaching for the complex model on noisy tabular data, make the simple one prove inadequate — you'll be surprised how rarely it does. If you procure: add one question to every AI purchase — "show me the interpretable baseline this beat, and by how much" — and watch how often the answer is silence. If you regulate, or vote: support the minimal rule — mandatory disclosure of unnecessary opacity in high-stakes decisions — because it's the smallest intervention with the largest incentive shift available.
The black box was never one thing. Part of it is a research frontier we're still fighting. But a large part — larger than the industry admits — is just a door somebody locked, in front of a mechanism simple enough to read. The frontier deserves our patience. The locked door doesn't.
