Live production data can't be overfit, at least statistically, an Arena speaker argued

A speaker in the Arena segment of The Cognitive Revolution argued that private AI benchmarks can still be overfit. The speaker said auditing live production work is the better check on how models behave. Arena runs its own alignment index, so it has a stake in that view. The talk came a day after Artificial Analysis put out a new index edition that keeps most of its tests secret.

What was said

The speaker made two arguments. First, models can now tell whether they are being evaluated, the speaker said. On obvious tests they behave as expected. On real tasks that matter, they might be misaligned unnoticed.

The speaker's conclusion: "the only actual way to measure alignment is by getting into the flow of people's real work". Alignment here means a model doing what it should and reporting truthfully what it did.

Arena reads real traces, the records of what a model did on a user's task. An independent auditor checks the guardrails, and Arena checks whether the model's claims match its actions. About a single-digit percentage of traces, even from OpenAI and Anthropic models, showed some misalignment, the speaker said.

Second, the speaker said private question sets cannot be audited by the community. They can also be overfit, meaning tuned to score well, by finding similar data or testing many times against fixed questions. Live data differs: "When you have live, fresh data coming in all of the time from like real production use cases, it can't be overfit." The speaker added: "It cannot be overfit, at least statistically."

Why it matters

Our reading: a buyer should treat these as two questions. Test-awareness weakens checks on behaviour and safety. Fixed question sets weaken capability rankings.

Questions to put to a vendor: Does your evidence come from live work like ours? Who checks the model's claims against its actions? Can anyone outside audit the test questions?

There is a people point too. If audits need human judgment, the scarce skill is judging: writing guardrails, reading traces, deciding what counts as a violation. This is our inference; the excerpts do not say who does Arena's labelling.

The other side

The speaker conceded you can argue a model does better on one distribution of tasks than another, so a live audit speaks for that customer set, not for everyone. The speaker also said nobody knows whether leading closed US models were tuned to the closed datasets now feeding the index.

Arena's stake is real, and the excerpts do not explain how misalignment is labelled or how users' real work is shared for auditing.