Here's a pattern I've watched play out across a decade of machine learning adoption, and once you see it you can't unsee it. When a model classifies photos of pets, nobody demands to know its reasoning — if the accuracy number is high, everyone's happy. The moment the same class of technology starts predicting whether a tumor is malignant, whether a loan gets approved, whether a person walks free — suddenly everyone in the room wants to know why the model said that. And they discover, often for the first time, that nobody can tell them.
I've stopped thinking of explainability as a static property that systems either have or lack. It's better understood as a demand curve: the required depth of explanation rises with the stakes of the decision. The industry's problem is that our systems were built at the bottom of that curve — in the benchmark-and-leaderboard world where "why" never mattered — and are now being deployed at the top of it, where "why" is the whole question.
How we got an explainability deficit
The gap didn't appear overnight; it compounded. There's a long-standing tension in model design: pile on complexity — interactions, compositions, features multiplied together — and you can squeeze out more performance, at the cost of no longer being able to trace what any individual factor is doing. For years, that trade looked free, because the tasks were low-stakes and the metric was king. Then models grew to millions and billions of parameters, and interpretability didn't just lag — it fell generations behind. We now have a global industry deploying decision-makers whose decisions nobody can narrate, into domains where narration is legally, ethically, and practically mandatory.
The bill for that accumulated deficit is arriving in a form executives actually feel: stalled adoption.
The adoption bottleneck nobody budgets for
Talk to the people trying to bring AI into specialist domains — medicine, engineering, science — and you hear the same story with different nouns. The model performs well on the retrospective evaluation. The pilot gets built. And then the domain experts... decline to use it. Not out of technophobia — out of a completely rational position: if I can't see what this system is doing, I can't see why it's better than what I already do.
Sit with that sentence, because it's the crux. A domain expert isn't an empty slot waiting for an algorithm. They have a working method, hard-won judgment, and personal accountability for outcomes. Asking them to swap their judgment for an unexplained score isn't asking them to "adopt a tool" — it's asking them to take on risk they can't assess. Refusing is what a responsible professional should do. I'd go further: the expert who refuses an unexplainable model is applying better decision theory than the vendor selling it.
This reframes explainability economics entirely. Explanation isn't a compliance garnish added for regulators. It's the adoption mechanism. An explanation is how a model's competence becomes visible to the person whose confidence decides whether the system gets used at all. Understanding why the model recommends this antibiotic, this maintenance window, this classification — that's what converts a skeptical expert into a user. No explanation, no trust; no trust, no deployment; no deployment, and your model's accuracy might as well be zero.
The test most explanations fail
But here's where I've grown wary of the boom in explainability tooling: most of what gets produced under that banner explains to nobody for nothing. Saliency maps admired in demos. Feature-importance charts that require an ML degree to misread correctly. Artifacts generated because the checklist said "explainability," consumed by no one, changing nothing.
The test that separates real explanation from explanation theater is brutally simple: did it change what someone did? An explanation earns its existence when it helps a person make a different, better decision — when the clinician, seeing the reasoning, catches the case where the model missed context; when the engineer, seeing which signal drove the prediction, notices the sensor drift; when the loan officer, seeing the factors, spots the one that shouldn't be there. Explanation that supports action is infrastructure. Everything else is decoration.
This is also why the most interesting current direction — using language models to translate model internals into plain, readable prose — is both genuinely promising and genuinely dangerous. Promising, because it finally collapses the expertise barrier: explanations non-specialists can actually read, in their own vocabulary. Dangerous, because readability and faithfulness are different properties, and a fluent narrative about what a model "was thinking" can be wrong about the model while sounding wonderfully right. (That problem deserves its own essay.) The direction that matters is explanations designed around the receiving human — their vocabulary, their decision, their moment of use — with the faithfulness question treated as seriously as the fluency.
Build for the top of the curve
The practical conclusion I've landed on: since the stakes gradient only moves one way — AI keeps climbing into more consequential decisions — teams should build for the demand curve's top even when launching at its bottom. Concretely: know who must be convinced by your system's reasoning and design the explanation for that person, not for your ML team. Prefer architectures whose reasoning can be surfaced honestly over post-hoc stories where you can afford to. Measure explanations the way you'd measure any feature — by whether the humans downstream used them and acted differently. And treat every domain expert who says "I don't see why this is better than what I do" not as a change-management obstacle, but as your most valuable reviewer — the one telling you exactly what your product is missing.
The era of nobody asking why is ending, one high-stakes deployment at a time. The systems that thrive in what comes next won't be the ones with the best benchmark scores. They'll be the ones that can answer the question.
