The AI conversation has a strange shape right now. We argue endlessly about capabilities — what the newest model can and can't do — and comparatively little about the plumbing underneath consequential deployments: who answers for the decision, where the data came from, and whether anyone can actually see how the system works. Having spent a lot of time around AI in genuinely high-stakes settings — medicine, infrastructure, criminal justice — I've come to believe those unglamorous questions decide almost everything, and that we keep getting three of them wrong.
1. In high-stakes decisions, AI should assist — and a human should answer
Start with accountability, because it clarifies everything downstream.
For decisions that seriously affect a person's life — a diagnosis, a sentence, a loan, the shutdown of critical infrastructure — I hold a position that sounds conservative until you sit with it: the decision should be made by a human, and the AI should be an aid. Full delegation to a machine is defensible only in narrow circumstances: genuine time pressure that humans can't match, in situations where the machine demonstrably does far better. Those cases exist. They are rarer than the industry pretends.
The reason isn't sentimentality about human judgment. It's that accountability has to rest <em>somewhere</em>, and it cannot rest with a model. A radiologist assisted by a model that highlights a suspicious region is still the accountable party — and that arrangement only works if the assistance is legible to her. She cannot meaningfully accept responsibility for a recommendation she cannot interrogate. This is why, in high-stakes work, interpretability isn't a preference; it's what makes the accountability chain real instead of theatrical. "The algorithm said so" is where accountability goes to die.
And legibility earns something money can't buy: legitimate trust. The most widely adopted clinical decision tools aren't sprawling neural networks — they're small transparent scoring systems that clinicians can inspect, question, and <em>own</em>. Meanwhile, regulators have approved plenty of black-box medical models that later simply didn't work in practice. Opacity didn't just make those failures harder to catch; it made them harder to even define. A transparent model that's wrong gets caught by the experts using it. An opaque model that's wrong gets discovered by its victims.
2. The real bottleneck isn't algorithms — it's public data
Here's the constraint almost nobody outside the field appreciates: for many of the highest-stakes applications, the limiting factor isn't model architecture. It's that the data needed to build and honestly evaluate models is locked up.
Consider health monitoring from wearables. Hundreds of millions of wrists now carry sensors streaming cardiac signals. The potential public-health value of great models over that data is enormous. But the only people who can build them are the handful of companies that own the devices — because the large datasets are proprietary and the public ones are, frankly, poor. Everyone else, including most of the world's research talent, is locked out. The data is the moat, and the moat is holding back the field.
There's a proven fix, and it's beautifully boring: public benchmark datasets, ideally curated by governments or standards bodies. We've watched this movie before. When a national standards institute builds a serious dataset and runs an open evaluation challenge, the quality of an entire field's algorithms rises — it happened dramatically with facial recognition, where a public benchmark became the driving force behind global progress. There's no reason the same lever can't be pulled for cardiac signals, EEGs, and a dozen other domains where better models would translate directly into lives.
I'd add one principle on top: models built for the public should be <em>owned</em> by the public. Published, inspectable, usable — not proprietary artifacts extracted from proprietary data. A company that collected data expensively will never release it; that's their secret sauce, and expecting otherwise is naive. Which is exactly why public data infrastructure is a job for institutions whose mission is the commons.
3. Nobody knows how to make an LLM interpretable — and we should say so
Now the uncomfortable one. The obvious question of this AI moment is: can large language models be made interpretable? The honest answer, from everything I can see, is that <em>nobody knows how</em>.
Yes, there are entire research communities probing the insides of these models — and that work is worthwhile. But let's be precise about what it is: poking at the internals of a black box to guess what it might be doing. That is explanation-of-a-black-box, not interpretability. An interpretable model is built under constraints that make it understandable <em>by construction</em>. Nobody currently knows how to build something with an LLM's capabilities that way. It took the field years to get from breakthrough image classifiers to genuinely interpretable computer vision models — and language models are a harder, stranger object. Compounding the problem, meaningful experimentation on frontier models is only possible inside a few companies with the compute and access to run it. The rest of the world can't even study the question properly.
This doesn't mean "never use LLMs." It means matching the tool to the stakes, honestly. There's promising work on agentic systems that, when several tools solve a task equally well, deliberately prefer the most interpretable and reliable one — pushing transparency into the parts of the system where we know how to have it. That's the right instinct: interpretability where it's achievable, honesty about where it isn't, and real hesitation about wiring an uninterpretable component into decisions where someone's life, liberty, or livelihood rides on the output.
The thread connecting all three
Accountability, data, and honesty about opacity aren't three separate policy debates. They're one debate. Human accountability is only real when systems are legible. Legible systems can only be built and vetted when data is open enough for independent scrutiny. And the scrutiny only means something if we're honest about which systems can be understood and which, for now, cannot.
The capabilities race will take care of itself; there's no shortage of money or talent chasing it. The plumbing is what actually decides whether AI in high-stakes domains earns trust or merely demands it. And trust that's demanded rather than earned has a way of being withdrawn — usually right after the failure nobody could see coming, inside the box nobody could see into.
