There's an assumption baked so deeply into how our industry builds AI that most people no longer notice they're making it: <em>if you want the most accurate model, you have to give up on understanding it.</em> Complexity buys performance; transparency is the tax you pay for it. Pick one.
I used to believe this. I don't anymore, and the reason is simple: for a huge class of real-world problems, it's empirically false. And because it's false, an enormous amount of the opacity we tolerate in consequential systems — models that influence who gets hired, who gets a loan, how patients are triaged — is opacity we chose, not opacity we needed.
First, let's get the vocabulary right
Before going further, a distinction that the industry blurs constantly, and that I've come to think matters more than almost any other in this space: <strong>interpretability is not explainability.</strong>
An <em>interpretable</em> model is one a person can actually understand — the model itself is the explanation. Its reasoning is inspectable end to end, because it was built under constraints that keep it comprehensible.
<em>Explainability</em> is something else entirely: taking a black box you don't understand and generating post-hoc stories about what it might be doing. Feature importance charts, saliency maps, "the model focused here" heat maps. These are pokes at the outside of a box. They are approximations of the model's behavior, not the model's behavior — and an explanation that's only approximately faithful is precisely the kind of thing you shouldn't hang a high-stakes decision on.
There's now a whole industry selling explanations of black boxes as if they were interpretability. Be skeptical. If the explanation isn't the model, you're trusting two things: the model, and the story about the model. That's more faith, not less.
Where the myth breaks down
Here's the part that surprises people. The accuracy-requires-complexity intuition comes from domains with enormous, relatively clean datasets — image recognition being the canonical case. Ten photos containing a chair either have a chair or they don't; the signal is crisp, and deep models feast on it.
But most consequential decisions don't live in that world. They live in the world of <em>noisy</em> data: medical records, financial histories, human behavior. Take two patients with identical charts today — one may have a stroke next year and the other may not. The relationship between inputs and outcomes is fundamentally non-deterministic. There's a ceiling on how much signal exists, and no amount of architectural complexity buys you through it. Past a point, complexity doesn't capture more truth; it captures more noise. It overfits.
And in exactly these domains — the noisy, high-stakes ones — a striking pattern shows up again and again: simple, constrained, interpretable models perform about as well as black boxes. Predicting whether someone released from prison will be re-arrested is a famous example. Nobody can predict that well, because nobody knows what the next three years of a person's life hold. Black boxes do roughly as well as simple models — which means for a decision about a human being's freedom, there is no accuracy justification for the black box at all. You're giving up transparency and getting nothing back.
The model that fits on an index card
My favorite existence proof is the humble medical scoring system. Some of the most widely deployed clinical decision tools in the world are models you can write on an index card: a few factors from a patient's history, two points for this, three points for that, sum it up, read off a risk. Clinicians in intensive care units use tools like this daily — including for problems as serious as estimating seizure risk in critically ill patients from a handful of EEG features. A model small enough to memorize, doing frontline work in life-and-death settings, at accuracy competitive with anything opaque.
And there's a version of this story that I find even more instructive. A celebrated deep-learning radiology model — a genuine black box, convolutions and transformers all the way down — was widely believed to be doing something no interpretable model could replicate: predicting breast cancer risk years in advance from mammograms. When researchers finally dug into what it was actually keying on, the answer turned out to be <em>subtle asymmetries between the left and right breast</em>. Once you know that, you don't need the transformer. You can build a symmetry detector — a model that is understandable, and just as accurate, and <em>actionable</em>, because now you can point at the exact place in the image driving the risk score. The black box wasn't doing magic. It was doing something simple, expensively hidden.
Interpretability is a debugging tool, not a luxury
Why does this matter so much in practice? Because real-world models are built from real-world data, and real-world data is a mess. Sensors mislabel. Databases lie. Pipelines drift. The single most valuable property a model can have in that environment is that the people who understand the <em>domain</em> — engineers, doctors, analysts — can look at the model and tell you when it's wrong.
I've seen the pattern repeatedly: a black box trained on messy operational data produces confidently absurd recommendations, and nobody can tell whether the problem is the data, the features, or the model, because nobody can see inside. Make the model interpretable, and suddenly domain experts become your debuggers. They spot the artifact, the leakage, the upstream data problem. Accuracy <em>rises</em> — not despite the interpretability, but because of it.
That reframes the whole trade-off. The question isn't "how much accuracy will I pay for transparency?" For noisy, high-stakes problems, the honest answer is frequently: nothing. The question is what you're getting for the opacity — and too often, the answer is a system nobody can troubleshoot, a vendor's moat, or a story that "sophistication" sells better than simplicity.
So here's the rule I now apply before reaching for a black box: try the simple, interpretable thing first, and <em>make the complex thing earn its opacity</em>. If the black box can't beat an index card, the index card wins. In high-stakes decisions, it should win even when the race is close — because when the model is wrong about a human being, "we can see exactly why" is not a nice-to-have. It's the whole game.
