Skip to content
Security & Trust Talking point

Adam Shostack: AI that is usually right can erode its reviewers' vigilance

Adam Shostack told Black Hat that frequent correct answers make it hard to stay alert to the rare miss.

W
WebPulse Newsroom
AI-assisted · 2 min read
Share on X LinkedIn
Adam Shostack: AI that is usually right can erode its reviewers' vigilance
In brief
  • Adam Shostack said LLMs are right often enough that people lose vigilance, and staying alert is hard when most checking turns up nothing.
  • In his bank story, a lawyer caught a contract clause the LLM missed, and he could not get the model to explain the miss well.

Adam Shostack, founder of Shostack + Associates, argued on the Black Hat show that an AI tool that is usually right can wear down the people who check it. He spoke at Black Hat USA 2026. When errors are rare, he said, staying alert is hard.

What was said

Shostack described over-reliance on LLMs, one of eight threats in PHANTOM-B, his threat modeling checklist for LLM products. People trust the model, he said, and do not watch closely what it does. These tools produce a lot of code and text.

In his training class the day before, a student said the model had given 50 pages and they had not read them yet. Shostack said the volume is hard to focus on. It is worse because "it is right often enough that we lose vigilance." Staying alert is hard when most of the effort is wasted.

He also described how these tools work: "They're token generators. They come up with statistically likely next things." In plain words, the model writes the text most likely to come next, one small piece at a time.

Earlier in the talk, in a section on explainability, he told a bank story. He was proposing work to a bank. The bank's contract said nonpayment of invoices was not a breach. In his words, "my lawyer caught it, but the LLM didn't." He then tried to make the model explain the miss and write a prompt that would catch it. He said he failed. Its best explanation was that the clause sat in the middle of a long paragraph that read like boilerplate.

Why it matters

Our reading: fluent output is hard to audit. A model that predicts likely text may glide past a clause that looks routine. Asked why, it gave only a loose reason after the fact. The bank story also shows a check that worked, because a lawyer actually read the paragraph. Set that against 50 unread pages. For managers, the question is what a reviewer really reads before approving. Shostack said that when it is your code and your brand, the choice is yours.

The other side

Shostack did not compare a mostly right AI with one that fails loudly. That comparison is not his. He gave no figure for how often models are right, and the excerpts offer no fix for fading attention. The bank story was about explainability, and it is one case. He also said decisions on hiring or college admissions are different, because other people bear the cost. He cited an Ontario system for deciding who went to solitary, which he said was biased against Black people.

Written by the WebPulse Newsroom with AI assistance, and checked by our editorial review: every quotation was verified against the recording's transcript. How we use AI.

The conversation this talking point comes from