Something quietly momentous happened to AI explainability, and I don't think its double edge has fully registered yet. For most of machine learning's history, understanding why a model did something required real technical expertise — reading attribution scores, interpreting heat maps, reasoning about gradients. Explanations existed, but only specialists could consume them. Then language models arrived, and suddenly it costs nothing to have a system narrate its own behavior in fluent, confident, perfectly readable prose. Ask "why?" and you get an answer a hospital administrator, a judge, or your grandmother can understand.
This is a genuine breakthrough in accessibility. It's also, I've come to believe, the most seductive trap in applied AI right now. Because the explanation you just read is also a model output — generated by the same kind of stochastic machinery, subject to the same fabrication tendencies, and checked, in most deployments, by absolutely no one.
Readable is not faithful
Keep two properties rigorously apart. An explanation is readable if a human can follow it. It's faithful if it accurately reflects what the model actually did — the real computational path from input to output. These properties have no necessary connection. A gradient-based attribution can be faithful and unreadable. An LLM's breezy narrative about which factors "led to this assessment" can be maximally readable and bear no reliable relationship to the mechanism it claims to describe.
The trap is that humans use readability as a proxy for truth. A fluent, structured, plausible account feels verified — that's what fluency does to us. So the better our explanation-generators get at prose, the more effectively they can launder an unfaithful account into confidence. We've built machines that are excellent at answering "why?" — and largely indifferent to whether the answer is true. In the low-stakes world that's a curiosity. In the world where explanations underwrite diagnoses, loan denials, and sentencing inputs, it's a mechanism for manufacturing unearned trust — explanation as anesthetic.
Red flags you can actually watch for
Short of solving faithfulness — an open research problem — there's a surprisingly practical signal: consistency under interrogation. Ask a model to explain the same decision twice. Rephrase the question. Probe the same case from a different angle tomorrow. When the "reasoning" flips — when the story cheerfully reorganizes itself while the decision stands still — you've learned something important: the narrative isn't causally tethered to the decision. It's decoration, composed on demand. A faithful account of a fixed mechanism shouldn't be this negotiable.
I'd encode that as operational doctrine: instability of explanation is a defect of the system, not a quirk of phrasing. Log explanations. Re-elicit them. Diff them. Treat a flip the way you'd treat a failing assertion — as a red flag that halts, not a wording issue that amuses. Most organizations currently test their models' decisions and let the explanations float free, unversioned and unaudited. That's backwards precisely where explanations are doing the trust-building work.
From vibes to guarantees
The deeper fix is to stop treating explanation quality as a matter of impression and start demanding the thing every other engineering discipline demands: guarantees. Not "this explanation seems reasonable," but provable properties — statistical or formal — connecting the explanation to the model's actual behavior. Does deleting the factors the explanation cites actually change the output? Does the explanation predict how the model behaves on perturbed inputs? Can we bound, with stated confidence, how often the narrated reasoning matches the mechanism? An explanation that comes with a verified error rate is evidence; an explanation that comes with nothing is content.
This is where the field's serious energy is heading, and none too soon — alongside pushes for standardized evaluation frameworks (so "our system is explainable" becomes a testable claim rather than a brochure adjective), interactive explanations that let users interrogate rather than passively receive, and explanations adapted to the receiving human — the same decision explained differently, and appropriately, to the surgeon, the patient, and the auditor. The common thread: explanation as an engineered, evaluated artifact rather than an act of theater.
The bar that matters: does it change what you do?
One more filter, and it's the one I apply first. Plenty of faithful explanations are still useless — technically true accounts that no human can act on. The explanations worth engineering are decision-relevant: they help someone do something differently. The clinician who sees the model leaned on an artifact and overrides it. The engineer who sees which signal dominated and finds the drifted sensor. The applicant who learns which factor sank the application and what would genuinely change the outcome. Explanation that never touches a decision is a museum exhibit, whatever its fidelity.
So the full bar, as I'd state it: an explanation should be readable by its actual audience, faithful to the actual mechanism, stable under interrogation, ideally guaranteed within stated bounds — and useful enough to change an action. Today's tooling gives us the first property nearly for free and leaves the rest as exercises. The corner-cutting temptation is enormous, because audiences reward fluency instantly and punish unfaithfulness only after the harm.
Here's the summary I keep returning to. We spent years worrying about whether we could trust the model's decisions. Now every model comes with a persuasive spokesperson, and we've inherited a second problem wearing the costume of a solution: whether we can trust the model's account of itself. Two systems now face verification — the decider and the explainer — and the second one talks better. Check it accordingly.
