Skip to content
The AI-First Web

Endor Labs: Fable 5.2's test run cost 72% less than 5.1's, 63 answers recalled

Security scores held flat once memorized answers were removed. A pass on a known flaw may not show skill.

W
WebPulse Newsroom
AI-assisted · 4 min read
Share on X LinkedIn
Endor Labs: Fable 5.2's test run cost 72% less than 5.1's, 63 answers recalled
In brief
  • Endor Labs ran Claude Code with Fable 5.2 on its coding tasks. Security matched Fable 5.1 at 36.9%, and the run cost $187 against $672.
  • Endor Labs confirmed 63 solutions as recalled from training, over three times Fable 5.1's count. Removing them cut functional passes from 94.4% to 64.8%.
  • Ask vendors whether a benchmark pass on a published flaw was reasoned or recalled, and test AI coding tools on your own private code.

A pass is not always a skill

When an AI tool fixes a known vulnerability, a buyer assumes it understood the problem. Endor Labs' latest results show why that assumption needs checking. A correct answer can come from reasoning, or from having seen the answer before.

That distinction matters when you pay for AI-written code. A tool that recalls published fixes may look strong on public flaws and still be untested on your own private code. This is the publication's view, built on one vendor's test.

What Endor Labs measured

Endor Labs ran Claude Code with Anthropic's Fable 5.2 on the same tasks it uses for its Agent Security League. According to Endor Labs, the model reached customers on November 9, 2026, priced 20% below the 5.1 generation.

Each task sits inside a real project and touches code that was once part of a security fix. The tool is never told this. Its only instruction is to follow good security practice.

Two scores result. FuncPass is the share of patches that clear the functional tests. SecPass adds hidden security tests taken from the original fix, so a secure patch must first work.

Endor Labs also removes any answer it confirms as copied. That includes fixes recovered from git history, the web, or training recall. A pipeline flags suspect cases and a language model judges each one.

Cheaper and faster, but not more secure

Fable 5.2 scored 36.9% SecPass and 64.8% FuncPass, according to Endor Labs. SecPass matches Fable 5.1 exactly. Codex with GPT-6 Astra scored 34.6%.

The cost of the run fell sharply. Endor Labs says the full run cost $187, against $672 for Fable 5.1. The median task took 1.9 minutes.

Most of that saving did not come from the price cut. Endor Labs calculates that the lower rates account for $47 of the $485 saved. The rest comes from the model using far less. Its output tokens, the units of text it generates, fell 68% from Fable 5.1.

Per secure pass that counted, Endor Labs puts the cost at $2.84, down from $10.19. An analyst quoted by InfoWorld had warned that Anthropic "has cut the price of a token, not the price of a thought". On these tasks, the thought also got cheaper.

The memorization problem

Endor Labs confirmed 63 Fable 5.2 solutions as recalled from training. Fable 5.1 had 18, so the new model has more than three times as many. Codex with GPT-6 Astra had none removed.

Removing them moves FuncPass from 94.4% to 64.8%. Fable 5.1 sits at 86.6% on the same basis.

The counted result would look very different otherwise. With recalled answers kept, Endor Labs says Fable 5.2 reaches 60.3% SecPass, 19 points above Fable 5.1. Of 53 recalled solutions inside the scored set, 42 had passed the security tests. After removal, the two models tie.

Endor Labs found that 47 of the 63 were new compared with Fable 5.1. Fable 5.1 had passed the functional tests on all 47 without a cheating finding. On those tasks, the older model had worked out its own answer, while Fable 5.2 reproduced the original project's fix.

Why this matters for published flaws

Anthropic says its sibling model, Mythos 5.2, rediscovered 89.6% of 1,829 vulnerabilities that had already been disclosed, working from code as it stood before the fix. Endor Labs did not test Mythos and measures writing secure code, not finding flaws.

Endor Labs still urges caution. Its own tasks are built from similar pre-fix code, and Fable 5.2's patches cleared the hidden security tests on 117 of them. For 48 of those, the fix was confirmed as reproduced from training data.

Endor Labs' takeaway is that on published CVEs, a plain pass and a pass reached independently measure different things. One task did show an independent solve. It maps to CVE-2022-3277, a resource-consumption flaw in OpenStack Neutron, and only Fable 5.2 passed its hidden tests.

Limits of the test

Endor Labs ran one harness at the default effort setting on its own tasks. Harness versions differed across the three Fable runs. Endor Labs also notes that the model wrote far less visible reasoning, with 36 of 1,361 thinking blocks containing text, against 1,098 of 4,264 for Fable 5.1.

Endor Labs says that makes edits harder to audit, because its anti-cheating checks must rebuild many edits from shell commands. Its test of Anthropic's other claims is mixed. It measured output about 19% faster, against the announced 30%, and saw no coding gain in its scores.

What to ask your team and vendors

First, ask vendors whether benchmark results exclude answers the model recalled from training. If they do not say, the score may measure memory.

Second, test AI coding tools on private code that has never been published. Recall cannot help there, so the result shows what the tool can reason through.

Third, check what the logs let you audit. If a tool leaves little reasoning and reads and writes through shell commands, ask how your reviewers will verify its work.

Cheaper code is useful. It only counts as cheaper secure code once you know how the answer was reached.

Produced by the WebPulse Newsroom with AI assistance from the original reporting credited below, and checked against that source by our editorial review. How we use AI.
Original reporting: Endor Labs.