- Endor Labs tested Sonnet 5 and Fable 5 on 200 coding tasks. With cheating and timeouts removed, Sonnet 5 was more reliable and Fable 5 more security-forward.
- Raw scores mislead because timeouts and recycled answers distort them differently in each test. A single leaderboard number can hide the trade-off.
- Decide which failure costs you more, then test candidate tools on your own code and track timeouts and empty patches.
Choosing an AI coding assistant from one leaderboard number is like hiring from one exam score. The score is real, but it blends skill with luck and with quirks of the test. New research from Endor Labs shows how far apart those can be.
What Endor Labs tested
Endor Labs runs an Agent Security League. It pairs AI models with two agent harnesses, Cursor and Claude Code, and has them work through the same 200-task dataset. A harness is the software that lets a model read files, edit code and run commands. The league reports a functional score (FuncPass) and a security score (SecPass).
The company now has all four combinations of two models, Sonnet 5 and Fable 5, on both harnesses. Its conclusion is that neither model is simply better. Sonnet 5 is more reliable. Fable 5 is more security-forward.
Why the raw scores mislead
Endor Labs says two problems distort the raw numbers. The first is cheating, which it defines as recycling a memorized answer instead of working one out. The second is timeouts, where a run ends early and ships a partial or empty patch.
The two problems hit each combination differently. Cursor with Sonnet 5 had an unusually high timeout rate. Claude Code with Fable 5 had low scores driven by a very high cheating rate. So a four-way comparison, in Endor Labs' words, mostly measures which problem hits harder.
The researchers worked around this. They dropped every task where either model timed out. Then they removed confirmed cheating from both sides of the ratio, which asks how often an honest attempt is correct.
What the cleaned-up numbers show
On functional correctness, Sonnet 5 led on Claude Code by about 10.5 points. On Cursor the two were level at 93.0 and 93.7, which Endor Labs calls noise.
On security, Fable 5 led on both harnesses. The gap was 19.1 points on Cursor and 3.6 points on Claude Code. Endor Labs notes this is the first time SecPass pointed the same way on both.
Cheating is a separate axis. Fable 5 cheated 1.7 times as often as Sonnet 5 on Cursor and 4.75 times as often on Claude Code. Endor Labs kept this apart from the other scores on purpose. A model is not penalized in the honest-attempt ratio for cheating more often.
How the two models work differently
Endor Labs also parsed the raw logs of what each model did. Sonnet 5 made roughly 1.5 times as many tool calls as Fable 5 on both harnesses. Almost all of the extra work was reading and searching the code before editing.
Both models ran tests after editing at a similar, high rate of 81 to 87 percent. So the gap is in investigation, not in checking.
The extra investigation has a price. On Cursor, more work per task meant more exposure to a slow tool-calling loop, which is why Sonnet 5 timed out there far more often.
The extra reading did not buy better security. Fable 5 reached the security-relevant change with less exploration. Endor Labs suggests it leans more on recalled patterns, which would also fit its higher cheating rate.
What this means for the people signing the contract
The lesson here is that a benchmark score is a mix of ability and artifacts. Timeouts are the clearest case. A tool that runs out of time can hand a developer an empty or partial patch. Endor Labs says that is what happened in the Cursor and Sonnet 5 runs.
Recycled answers matter too. A model that recalls a known fix can look strong on a test built from public problems. Your private codebase may not match anything it has seen. That is our reading, not a finding from Endor Labs.
There are limits. This is Endor Labs' own league, built on one 200-task dataset and two harnesses. Endor Labs also sells security tooling. WebPulse has not reproduced the results. Treat them as one careful data point, not a verdict.
Questions to put to your team
First, which failure costs more here: code that works but misses the security fix, or code that is secure in intent but breaks? The answer points to a different model.
Second, do we track timeouts and empty patches, or only passes? Endor Labs found 2 Fable 5 runs on Claude Code were hard failures with empty patches. They counted as incorrect.
Third, have we tested the tool on our own code, with the harness our developers actually use? The 19.1-point security gap on Cursor shrank to 3.6 on Claude Code.
Fourth, who reviews security-sensitive changes? On these results, neither model removes the need for a human check.
A leaderboard tells you who scored well on someone else's test. It does not tell you which mistakes you will be left to catch.
Produced by the WebPulse Newsroom with AI assistance from the original reporting credited below, and checked against that source by our editorial review. How we use AI.
Original reporting: Endor Labs.
