Ramp says cheaper models nearly match frontier ones at finding flaws when run as a swarm

Alexander Shevchenko of Ramp Labs said on the AI Engineer show that open-weight models up to five times cheaper per token found almost as many flaws when run as a swarm. He spoke only about Ramp testing itself. The risk for readers is a question: could an attacker use the same approach?

What was said

Shevchenko said Ramp used a cloud agent called Ramp Inspect to run 10,000 parallel agents against all of Ramp. The agents hunted for vulnerabilities and wrote tickets. Other models then removed duplicates. They also tried to reproduce each flaw, to check it was real and important. Ramp found seven high-severity vulnerabilities and patched them.

One flaw was a script injection that could arrive by email. Shevchenko said "no single component of that stack was unsafe, and the kind of swarm had to patch together multiple components". In his telling, the swarm had to understand several parts end to end.

He also described the cost. He said the run is automatic and "You don't need to spend anyone's time." The cost, he said, is the tokens, the units AI models charge for. He said "you can actually use open-weight models that are up to five times cheaper compared to the frontier reasoning." Open-weight models are ones anyone can download.

He said that because it runs as a swarm, you can spend more on the cheaper tokens and still get almost as good a detection rate. He did not say Ramp did so.

Why it matters

Our reading: Shevchenko described a defensive tool, but nothing in his method is limited to defenders. The swarm needs no one's time, and its running cost is tokens. The cost story has a gap, though. Price per token falls, the number of tokens needed rises, and no net cost was given.

So the attacker risk is a question to ask, not a conclusion. An attacker would also need to build what Ramp Inspect provided, the setup for running thousands of agents. Ramp also knew its own systems.

One of Ramp's seven flaws only appeared when several components were read together. Teams that buy or run software may want to ask whether anyone has tested the whole chain.

The other side

Shevchenko spoke about Ramp's own testing. He did not discuss attackers. He gave no total cost for the run, and he did not say how close the cheaper models came.

The result also comes from one company's article about its own systems. The excerpts do not say how well the approach works on other software. They also do not say how many false alarms the swarm produced before the filtering steps removed them.