Jewkes says routing out ambiguous and unusual cases got critical tasks to about 95%

Matthew Jewkes of Standard Cybernetics argued on the AI Engineer show that sorting cases into errors, undecided cases and new ones did much of the work. He also said reading the evals behind it is costly, tiring work.

What was said

Jewkes described a Clarivate system that indexes drug development documents. The show notes say an early agent-based version scored in the low 80s. The team then worked through 1,000 evals with domain experts. An eval is a test case that checks the output.

Three problems appeared. The first was plain error. Jewkes said even top models fail 2% or 3% of the time in multi-step workflows.

The second was undecided cases. Two experts might confidently disagree, or the organization had not settled an answer. His example was whether colchicine, a gout drug, targets a certain molecule. He said two leading models each go back and forth on it.

The third was novelty. Four of six mechanisms of a heart disease drug candidate were new to science, so humans had to update the indexing rules.

Each kind got its own route. If the AI is confused or the case is unusual, the system records why and skips it. A plausibly business-critical case goes to a heavily tested pipeline. If that pipeline is still unsure, the case goes back to a human.

With ambiguous and unusual cases removed, Jewkes said standard approaches reached about 95% on the critical tasks. Getting from 95% to 99.95% was hill climbing through many evals. Looking back, he said routing took at least as much time as that final climb.

Why it matters

Our reading: a headline accuracy figure depends on which cases the system may hand back. A buyer should ask what share goes to humans, who reads that pile, and who decided the policy behind it.

Jewkes argued that routing helps experts. It removes some routine work from their jobs. It also leaves the remaining cases organized, so experts can give them structured attention.

The cost sits elsewhere in his account. His team had to read why the AI missed on each eval. He called it "exhausting work because you're reading sycophantic text and you're trying to figure out what specifically went wrong". He also said cheap execution raises the load on judgment, which he thinks is currently underpriced.

The other side

Jewkes did not call this cheap. He said full automation at this quality is expensive. Engineer time on evals costs money, and so does the climb from 95% to 99.95%. He would not build such systems for a startup still in discovery.

His claim that the system beat PhD-level humans at scale is his own, and he qualified it. A single expert on a single case would do better. He said the team deployed only after a random sample showed the system beat past human work.

The excerpts give no clear share for undecided cases. Jewkes gave about 3% to 5% for exceptions. How heavy the burden on experts is in other organizations remains unmeasured.