- In one team's test, every error sat in the few case families where two models disagreed, so disagreement showed where to look.
- Four model families still gave the same wrong total in a benchmark, because they all used a sign convention the answer key did not.
Agreement between AI models is not verification. Hrishi Olickel of Southbridge AI argued this in a talk on the AI Engineer show about using AI to transform data. In his team's test, every error sat where two models disagreed. Yet four model families still gave one wrong total.
What was said
Olickel described quiet failures on an Ohio campaign finance project. One pipeline searched for "in-kind" spelled with an underscore, but the file used a capital I and a hyphen. So 15,658 in-kind contributions were treated as regular money.
A second pipeline allowed for four date styles, but the file held a fifth. Some 99.4% of dates came out empty. Row counts and dollar totals still matched, so the agent moved on.
In a third case, several runs saw negative loan amounts and judged the sign redundant. They stored absolute values. In his words: "A net of $784,000 became $5.3 million of debt in the end, validated by multiple models."
Cross-checking helped. Two models were run on 200 held-out case families. They gave the same answer on 194, and not one of those matching answers was incorrect. Olickel said: "Every error sat in the six families where they disagreed." Separately, he remarked that models from different families tend to disagree more.
A benchmark showed the limit. Four model families landed on the identical total. All four used a sign rule that differed from the one in the answer key.
Why it matters
Our reading: a second model is a cheap way to find where to look. It is not a sign-off. The date failure passed because the checks were row counts and dollar totals, which stayed intact. The sign failure passed because every checker made the same assumption.
For anyone buying or running AI-driven data work, Olickel's advice was to define a failure surface before starting. That means a baseline, partly checked by hand, that results can fail against. Sign-offs on money fields should rest on something outside the models' shared habits.
The other side
Olickel conceded that cross-model checks only shrink the problem. What remains is a smaller but persistent class of errors that, he said, affects language models in general. His answer is to list those errors and apply specific fixes.
He also called the failure surface the hardest part to turn into tooling. He said the best current models are only barely good at it. Human review catches these errors too, but he called humans slow and expensive.
The figures come from his team's own projects. He did not say whether the two models that agreed on 194 came from different families. The talk does not show whether that result holds elsewhere.
Written by the WebPulse Newsroom with AI assistance, and checked by our editorial review: every quotation was verified against the recording's transcript. How we use AI.
The conversation this talking point comes from
- AI Engineer: Just-in-Time Ontologies: What Cheap Intelligence Means for Data — Hrishi Olickel, Southbridge AI (2026-10-12)
