- Microsoft and City University of Hong Kong researchers tested 14 models on long-task choices. The top model scored 59.7% on a strict test where guessing scores 25%.
- Average accuracy across all 14 models was 62.3% when the deciding clue was already visible, and 21% when it appeared only after extensive later work.
- Ask vendors for evidence on step-level judgment, not only task completion, and place human review at the points where the cost of a choice is delayed.
An agent that finishes a task has not necessarily made good choices along the way. A building can pass its final inspection and still rest on a poor foundation. The problem shows up later, when someone tries to change it.
A new benchmark makes this gap measurable for AI agents. Researchers at Microsoft and City University of Hong Kong use the word "taste" for good judgment about a choice whose effects have not yet appeared. Their test finds that current models are only moderately good at it.
What the researchers built
Most agent tests ask one question: did the task get done? Taste-Bench asks about the steps in between. The authors say existing benchmarks measure completion, not the quality of the choices along the way.
The test uses "decision forks." Agents often attempt the same task several times, and those attempts split at some point. The researchers freeze a recorded run at that split, hide everything that came after, and show a model two candidate next steps. The model must pick the better one.
No human expert decides which is better. The label comes from hindsight: the later work and outcome of each branch. The Decoder reports that human review confirmed almost all of the auto-generated labels.
Forks come in two kinds. A parallel fork comes from two separate attempts at one task that begin from the same starting point and then go different ways. A detour comes from one agent that wanders off the productive path and later fixes itself. For detours, the question is whether a model can see the error earlier than the agent did.
The 502 questions draw on software engineering tasks (SWE-bench and SWE-bench Pro) and machine-learning research transcripts from METR's public MALT release.
A deliberately strict score
Each question is asked twice, once with the options in the published order and once reversed. The model gets credit only if it answers correctly both times. Random guessing therefore scores 25%. A model that always favours one position scores zero.
That strictness matters when reading the results. The headline figure is not a casual hit rate. The authors also report a looser single-order average, but the sources here do not give its value.
Where judgment breaks down
The test covered 14 models. They came from seven families, among them Claude, GPT and Mistral. The top scorer, GPT-5.6 Sol, averaged 59.7%. GPT-5.5 followed at 59.5%. All other models scored well below that. Detour forks proved harder than parallel ones.
The sharper finding concerns timing. The team measured how far into the later work an observer would have to look before the better path became clear. When the key clue was already in the earlier steps, average accuracy across all models was 62.3%. When the clue appeared only after extensive later work, the average across all models fell to 21%. That is below the 25% guessing line.
Extra thinking did not close the gap. The researchers repeated the test at three reasoning settings for two models. Going from the least to the most reasoning effort left accuracy almost unchanged. Token use was highest on the hardest tier, yet longer thinking did not help.
What this shows, and what it does not
The benchmark points to a delegation risk. A poor choice and a good one can look the same for a long time, so a weak decision may not announce itself. That is our argument, not something the benchmark measures directly.
The data fits it in one respect. Where the deciding clue appeared only after extensive later work, average accuracy fell to 21%. That resembles real choices whose cost shows up late, though the benchmark does not measure real cost.
The limits are real. These are frozen, two-option questions, not live agent runs. They cover only coding and machine-learning research. The queries and candidate steps were model-generated from recorded trajectories. The benchmark shows how models judge forks in recorded runs. It does not show how often a deployed agent makes a costly choice in your environment.
What leaders should ask
First, ask vendors what evidence they have for step-level quality, not only task completion. A high completion rate can hide weak choices that were rescued later at extra cost.
Second, find where in your agent workflows the cost of a choice is delayed. Architecture decisions, which experiment to run next, and which code to build on are examples. Put human review at those points rather than only at the end.
Third, do not treat a bigger reasoning setting as a fix. In this test, more thinking did not help.
Fourth, ask your teams to keep run records. The benchmark works because recorded runs reveal, in hindsight, which branch was better. Organisations that keep such traces can start to measure judgment, not just output.
An agent that can finish the job is not yet an agent you can trust with the choices that decide how well it is done.
Produced by the WebPulse Newsroom with AI assistance from the original reporting credited below, and checked against that source by our editorial review. How we use AI.
Original reporting: Researchers from Microsoft and City University of Hong Kong (Taste-Bench authors, hosted on Hugging Face by wenbopan).
