- A Princeton-led paper argues a final score alone is not enough to support claims about AI agents, because evaluators must examine every step an agent took.
- Checking agents is costly and hard to do independently, and lab results may not carry over to real workplaces.
- Before buying or piloting an agent, ask whether the evidence covers the full trajectory, who ran it, and whether it matches your setting.
A vendor pitching an AI agent will usually lean on a benchmark score. A new paper argues that the score alone does not support the claims made about agents. The reason is simple. A score grades the result. An organisation that deploys an agent is trusting the process.
The paper is titled "Emerging Trends in AI Agent Evaluation." Its lead group at Princeton says it has worked on improving agent evaluation for about three years. Researchers from Stanford, the UK AI Security Institute and Brookings joined the effort. Authors Felix Chen, Stephan Rabanser, Sayash Kapoor and Arvind Narayanan summarised the findings on the AI Snake Oil blog on October 19.
AI developers have long backed claims about real-world performance with benchmark scores. The authors' central point is that it is now harder to tell whether the evidence supports the claims made about agents. Claims are easy to repeat. Checking them is the hard part.
Why agents are harder to grade
An agent uses tools and acts inside an environment. That changes what a test must examine. The authors write that the evaluator "must scrutinize the entire trajectory, not just the final outcome." The trajectory is the full sequence of steps the agent took.
One reason is shortcuts. The authors say it matters whether an agent took them, because agents have access to tools and environments. A high score reached by a shortcut says little about how the agent will behave in your systems.
A second problem is attribution. When an agent does better or worse, the authors say it is harder to tell which of its components caused the change. A vendor's improved score may not show what actually improved.
The checking is expensive
The authors say this work needs significant engineering effort and is getting more expensive. Their example is the ExploitBench benchmark. Testing several agents on it ran to tens of thousands of dollars in credits, they report. That figure leaves out the engineering effort to build and maintain the test setup.
Cost shapes who can check. The authors say independent evaluators increasingly depend on funding and on access to models. Access now includes reasoning traces, because frontier labs release summarized versions instead of raw ones.
The implication for buyers, in WebPulse's reading, is that if few can afford to test, much of the evidence they see may come from the people selling the product. The paper does not make that claim.
Even the checkers need checking
Agents generate large volumes of traces, the step-by-step records of what they did. The authors say evaluators struggle to inspect them at scale. The tools they use for this are, in the authors' words, not foolproof and themselves need evaluation.
The paper offers one case. After the recent OpenAI-HuggingFace incident, reviewers looked back at what happened by using other agents to spot harmful actions. The reviewers themselves acknowledged limitations and biases in that method. The lesson is that an AI monitor is a control you must test like any other.
A lab result is not a result in your office
The authors warn that a result at one scale may not carry over to another. An agent that shows skill in the lab can meet limits in a specific use case. They point to a METR randomized controlled trial. Despite expectations, using agent tools for software development increased the overall time to finish the tasks.
That is one trial on one kind of work, and it does not show that agents slow every team down. It does show that a capable agent and a faster team are different claims. The authors say the human effect has to be tested in context.
They also separate two questions. Can an agent automate a task? Or can it augment a person? Evaluation that asks only the first will show where human work can be replaced. It will not show how people could work better with the agent.
What leaders should ask
Before approving an agent purchase or pilot, put five questions to your team and the vendor.
First, does the evidence cover the whole trajectory, or only the final outcome? Second, who ran the evaluation, and did they have real access to the model and its traces? Third, was it tested in a setting like yours, with people like yours? Fourth, how were any automated monitors checked? Fifth, what would it cost to repeat the test?
The authors also name a gap. They say deployers and industry evaluators can supply real-world settings that lab work lacks. For public sector deployments, they say evaluation should be part of the process from day one. Your own pilot data can help fill that gap.
A score tells you what an agent produced. The trace and an in-context trial show how it got there.
Produced by the WebPulse Newsroom with AI assistance from the original reporting credited below, and checked against that source by our editorial review. How we use AI.
Original reporting: AI Snake Oil.
