- METR's November 2026 update finds frontier models beat most humans on two checkable task families, but not the best, and fall short on two judgment tasks.
- METR warns that evals like its own, cheap to build and score, may overlap with lab training environments. It stays uncertain how fast capabilities improve.
- Leaders should map which workflows have a cheap checker, test AI there first on their own tasks, and decide who owns the judgment calls the checker cannot make.
The skill that is easy to test is not the skill that is hard to find
METR's update shows steep progress on work where a computer can check the answer. It also shows a visible shortfall on two tasks where a person has to decide whether the answer is any good. METR is the Model Evaluation and Threat Research nonprofit.
This is the publication's reading: AI looks strongest where checking is cheap. METR does not say checkability alone explains the gap. The judgment tasks may also simply be harder. Still, where someone must exercise taste, a human may be doing the scarce work. That bears on what a leader buys and what a leader keeps.
What METR measured
In a post dated November 20, METR reports results for 15 models, from GPT-4o to the latest releases, Opus 5.5 and GPT-6 Astra. Earlier work rated models by the length of task they could finish, using the time a human needed as the yardstick. MIT Technology Review says that first plot showed this length doubling every four months.
The new task set adds longer and more open-ended tasks. On the checkable subset of the new tasks, METR puts the strongest public models at a 50% time horizon of up to 27 hours. That is the task length, measured by human expert time, that models complete about half the time. It does not mean models reliably finish 27-hour expert tasks. METR is open that this is only part of the picture.
Where models beat most people, but not the best
METR compared models with people in two task families that have large pools of existing human results. The first is 15 Al Zimmermann contests, which are multi-month optimization competitions with 97 to 413 entrants each. In the typical contest, the best models ranked at roughly the 96th to 98th percentile of entrants. About four humans still finished ahead of the best model in the median contest. METR says the top of the human pool is quite strong.
The second is a speed-training challenge called Budget NanoGPT. The best AI run gained 15.7% over the starting record in 48 hours. The author of one strong human record, helped by AI for coding, estimated about 60 hours for a 41% gain over the previous record. The two baselines differ, so read the gap as a rough guide to scale, not a head-to-head. METR concludes that top humans can still reach much larger gains than AI agents. It calls the comparison rough but informative. Each human record builds on earlier ones, and recent records can be AI-assisted.
METR also recruited expert human baseliners for three tasks: Train a Program, Minecraft Bot and Reverse Conway. Models beat those baseliners. METR adds a caveat. It guesses the baseliners probably did not match the skill of typical frontier AI researchers within their own domain. So that result may flatter the models.
METR also cites outside results. On MirrorCode, agents rebuild software and check their work against a full test suite. METR reports that Opus 5.5 solved 77% of the medium to large programs in that set. It describes these as jobs that would take human experts weeks to months working alone. On FrontierMath Tier 4, a set of research-level maths problems, GPT-6 Astra recently scored 98%.
Where the checker runs out
MIT Technology Review explains the likely mechanism. Models improve at verifiable tasks by attempting them many times and reinforcing what worked. This method, reinforcement learning, needs a clear signal of success. Coding and maths supply one. Choosing a research direction does not.
METR's two judgment tasks probe that gap. In the first, models designed new AI evaluations and METR staff scored them. MIT Technology Review reports that none was rated useful for the staff's research. In the second, models scored philosophical arguments and fell below human performance. METR notes that its own staff disagree about how relevant that task is. Two tasks are a small base for a broad claim.
A warning about the test itself
METR warns that evals like its own, which are cheap to build and score, may overlap with the environments labs use for training. It says this overlap may cause systematic measurement problems as reinforcement learning grows. Treat any vendor's benchmark claim with the same caution.
To probe this, METR built a rough proxy called Lifecycle Cost. It is the cost of building a task as a container, then running and scoring it 100 times. The costlier subset averaged 2.9 times the cost, and models did 2.9 times worse on it in time horizon, even at equal task length. Lifecycle Cost measures the cost of building and scoring a realistic task. It is related to, but not the same as, how cheaply an answer can be checked. METR calls its data noisy and says it is uncertain how the trend behaves at higher costs.
What leaders should ask
METR says current public evidence cannot rule out drastic acceleration of AI research in the next couple of years. It also says it is significantly uncertain how fast capabilities are improving. That is a reason to measure, not to guess.
Three questions follow for your teams. Which of our workflows have a cheap, automatic check, such as tests, reconciliations or rule engines? If METR's pattern carries over, these are the workflows to test first. METR tested research and coding tasks, not finance or operations work, so confirm on your own tasks. Who owns the judgment calls the checker cannot make, and are they still in the loop? And have we tested any vendor claim on our own tasks rather than on a public benchmark?
A model can rank in the top few percent of a contest and still produce an evaluation design that METR's own staff did not find useful. Plan for both.
Produced by the WebPulse Newsroom with AI assistance from the original reporting credited below, and checked against that source by our editorial review. How we use AI.
Original reporting: METR (Model Evaluation and Threat Research).





