- Averaged across two CREBench tasks, Frontkick used about half the tokens and scored nearly 3x higher, but Synacktiv calls it a technical draw, citing a prompt difference.
- Agent comparisons can measure the prompt and setup as much as the architecture, so one benchmark is weak evidence for a purchase.
- Ask vendors for tests run on aligned prompts, and ask what the agent can execute and who can audit it.
A winning score can measure the setup, not the design
When two AI agents compete, the winner may owe more of its lead to a small detail in the prompt than to a better architecture. That is the finding in a new test from Synacktiv, a security firm. The firm built one of the two agents, and it says so plainly in its own write-up.
The lesson for anyone buying or building AI agents is simple. A benchmark result is only as good as the match between the setups. Mismatched setups measure the wrapper as much as the model.
What Synacktiv tested
Synacktiv built Frontkick, an agent for Binary Ninja, a tool analysts use to take apart compiled software. It compared Frontkick with Rikugan, a free, open-source agent with 678 GitHub stars.
The test used two cases from CREBench, a benchmark for reverse engineering and cryptanalysis. In each case the agent gets a stripped binary, meaning a program with its descriptive names removed. Its job is to work out which cipher and key are in use, rebuild the decryption, and pull out a hidden flag.
The cases were AES-128-CBC and ChaCha20. Each ran four times, with a cap of 1M tokens per run. Both agents used the same model, Claude 5.5 Sonnet.
Averaged across both tasks, Frontkick used about half the tokens that Rikugan did. Its success score was nearly 3x higher.
How the two agents work
Both agents run an agentic loop. The model gets a task and a set of actions it can take. Each result is fed back to it, and it decides the next step until it judges the task done.
The two agents differ in how the model acts. Rikugan uses the tool approach. The model fills in a structured form, in JSON, to call a fixed menu of actions such as looking up a function. Synacktiv says this approach limits the attack surface, validates inputs, and leaves clean logs. It also makes it easy to stop a risky action, like a payment, before it runs.
Frontkick uses the CodeAct approach. The model writes Python code that runs directly inside Binary Ninja. One script can chain many steps, filter raw data and import libraries without prior declaration. Synacktiv says this saves round trips and avoids rewriting a tool wrapper each time the decompiler changes.
Frontkick also manages its memory. It keeps the four most recent turns intact and cuts older steps to 500 characters. It summarizes history when the context window fills. It retrieves verified code recipes when its own code fails.
Where the comparison breaks down
Synacktiv read the transcripts and found a bias. On AES-128-CBC, Rikugan kept trying to import a library, pycryptodome, that was not installed. It then rewrote the algorithm by hand, byte by byte. It never used the cryptography library that was installed.
Frontkick's system prompt includes an "Available libraries" block listing what is installed. Rikugan had no such list. It guessed.
On this one case, Rikugan used roughly 2.5 times the tokens of Frontkick. Synacktiv attributes the gap largely to the missing list of installed libraries. The figure is also close to the 1M-token cap, and it rests on four runs.
Two limits apply. The transcript review covers the AES-128-CBC case. The write-up does not explain the ChaCha20 result.
The library list is also a built-in mechanism of Frontkick, so it is arguably part of its design. Synacktiv still treats it as a prompt choice that Rikugan could adopt.
Synacktiv's conclusion is careful. It says the test does not show that CodeAct is better. It points instead to how much prompt choices matter. The firm calls the result a technical draw and says an even-ground rematch with aligned prompts is needed.
The trade-off behind the design choice
Synacktiv notes that large model providers favor the tool approach for security, governance and scale. Running arbitrary code for millions of users means isolating every session in costly sandboxes. Frontkick instead runs inside the analyst's own Binary Ninja Python environment.
This is the part executives should weigh. The more freedom an agent has to write and run code, the more there is to govern. The more it is boxed into a menu of actions, the more it depends on someone maintaining that menu. Neither choice is free.
What to ask your team or vendor
First, ask whether compared agents ran on the same prompts, the same model and the same environment details. In Synacktiv's test, a missing list of installed libraries coincided with roughly 2.5 times the tokens on one case.
Second, ask how many runs sit behind a result. This test used four runs per case on two cases. That is useful evidence, not a verdict.
Third, ask what the agent is allowed to execute, who can see what it ran, and where a person can stop it. Those answers decide risk more than a leaderboard does.
A benchmark tells you how a setup performed on one day. Before you pay for the agent, make sure you know which part you measured.
Produced by the WebPulse Newsroom with AI assistance from the original reporting credited below, and checked against that source by our editorial review. How we use AI.
Original reporting: Synacktiv.
