- Memento 3, built by a team including Huawei Noah's Ark Lab, finished all 25 public ARC-AGI-3 games on an unchanged Claude Opus 5, per Pandaily citing QbitAI.
- Pandaily credits the design around the model; the same model averages about 40.7 in the official harness. The public score does not show how it would do on hidden games.
- When judging AI agents, ask which model, which tests and which safeguards produced a score, and whether the agent keeps a readable, reversible record of its beliefs.
Buyers of AI tools usually ask which model sits inside. A result reported this week suggests a second question matters as much: what is built around the model. The model can stay exactly the same while the system around it changes the outcome.
What was reported
Memento 3 is an agent system from a joint effort that includes Huawei's Noah's Ark Lab and University College London. Pandaily reports on it, citing QbitAI. On the public games of ARC-AGI-3, the interactive benchmark from the ARC Prize, it completed all 25 games and 183 levels.
In this test, an agent is placed in strange, pixel-style games with no instructions. Nobody tells it the rules, the objective or the effect of each action. It has to find these out by experiment.
Memento 3 needed roughly 44% of the human action count. That earned it 100, the top mark, on a measure called Relative Human Action Efficiency (RHAE). The previous best public-set result came from an independent baseline by Sergey Rodionov. It also finished every game, scoring 99 with 8,347 actions.
The model did not change
The engine underneath is not a Huawei model. It is Claude Opus 5, set to its top reasoning level. The team left its weights untouched and did no training aimed at these games.
Pandaily concludes that the improvement came from the design built around the model. The 40.7 is an average in the official ARC Prize harness, which is a different setup from Memento 3's. The report does not say it was measured on the same public set. Treat the comparison as a pointer, not a controlled test.
Still, the idea is worth holding onto. Capability is not only bought by choosing a bigger model. It is also built in the loop that decides what the model sees, remembers and checks.
How the wrapper works
The design is called Code as Model. As it plays, the agent writes a rulebook in plain language. It then turns that rulebook into Python code that simulates the game. It plans its moves inside that simulator before acting.
When the real game disagrees, the agent rewrites both the rulebook and the code. In one game, it first guessed that a special floor mirrored the pattern it was carrying. It later found the floor actually turned the pattern 90 degrees clockwise.
A replay step keeps the code honest. The system goes back over its past moves and checks, frame by frame, that its simulator reproduces what really happened. Each frame is a 64 by 64 grid. Every edit is saved in Git, a version-control tool, so a faulty change can be undone.
This is close to a lab notebook. The agent writes down what it believes, tests the belief and keeps a history of corrections. A human can read that history.
Pairing two rival world models brought one game down from 899 actions to 597. Pandaily notes that only that game was tested.
Limits that matter
The public games leave a gap. A perfect score there does not show how the system would do on the hidden evaluation games, which are the real test of whether the skill carries over. Pandaily flags this point itself.
The authors concede that exploring burns many tokens, the units of text a model reads and writes. They also say messier real-world conditions have not been tried. WebPulse has not reviewed the paper itself. The figures here come from Pandaily's report.
One result points to a cost angle. The team also tried Atari Pong. From roughly 9,500 frames, the system worked out how the ball bounces. After that, its code ran the player alone, with no calls to the language model, and took three games by 21 points to zero. Once the rules lived in code, play needed no model calls.
What this means for an organisation
The benchmark is games, not business software. Nothing in the report says Memento 3 works on company systems. The method, though, mirrors what a person does with an undocumented system: probe, form a theory, test it, correct it.
That changes where an AI purchase should be judged. Two products built on the same model may differ, because the surrounding design differs. Read vendor score claims with that in mind.
Questions to put to a vendor or internal team:
1. Which model produced this score, and in which harness? A result in a custom wrapper is not the same as one in a standard test.
2. Were the tasks public or hidden? Only hidden tasks test generalization.
3. Does the agent keep a readable record of what it believes, and can its changes be rolled back?
4. What does exploration cost in tokens, and what part of the work can later run as plain code?
The lesson here is that the model is one ingredient. The loop that checks the model against reality is the part you can audit, and the part worth paying for.
Produced by the WebPulse Newsroom with AI assistance from the original reporting credited below, and checked against that source by our editorial review. How we use AI.
Original reporting: Pandaily.
