- Google Project Zero compared an AI tool and a fuzzer on Chrome's V8 engine. Overlap was about 25% in production reports and roughly 60% in a rediscovery experiment.
- Either way, a substantial share of bugs was found by only one approach, at least on this one target, Chrome's V8 engine.
- Project Zero advises vendors to use both fuzzing and agentic analysis, so more bugs are reported before users are affected.
Two tools, one engine, different blind spots
A common assumption is that AI bug finders will replace the older automated testing tools. Google's Project Zero has just published evidence that, for now, they do not.
The team compared two systems that Google runs on V8, the JavaScript engine inside Chrome. Fuzzilli is a generative fuzzer. It builds test programs and feeds them to the engine, hoping to cause a crash. BigSleep is an LLM-based system. It reads new code changes and writes a proof-of-concept test that crashes the engine.
The lesson here is simple. A security programme that retires one method for the other is not swapping tools. It is giving up coverage.
What the numbers show
Project Zero looked at six months, April to September 2026. That is when the automatic BigSleep pipeline went live. It counted only "actionable" vulnerability reports. Those are bugs not marked as duplicates or working as intended, and with at least one fix commit referencing them.
On that measure BigSleep found 45 vulnerabilities and Fuzzilli found 37. Project Zero calls it roughly a tie, with a small edge to BigSleep.
The overlap is the more useful number. Across the actionable vulnerability reports, 20 of a total of 81 were found by both systems. That is roughly 25%. Of those shared bugs, a narrow majority reached the bug tracker first through Fuzzilli, with BigSleep reporting them afterwards.
Why the overlap is hard to measure
Counting duplicates is messy. Project Zero describes a Fuzzilli bug that Clusterfuzz filed and closed on the same day, with no developer touching it. Most likely it duplicated another bug, but the crash signature differed slightly. Clusterfuzz is Google's triage system, and both tools report through it.
The researchers say this method tends to underestimate overlap. So they ran a second test. They took 112 historic Fuzzilli vulnerabilities from 2023 to mid-2026. For each, they gave BigSleep the code change that likely introduced the bug. They ran the agent up to 25 attempts per change.
BigSleep rediscovered about 62% of the original bugs. For about 12% of changes it found a different, since-fixed bug. For about 26% it found nothing. Project Zero puts overlap at roughly 60% on this test.
Caveats that limit the result
The two methods give very different overlap figures. Project Zero says that whichever is right, a substantial share of vulnerabilities was found by only one approach.
The experiment also tilts against BigSleep, the researchers note. They may have picked the wrong introducing change for some bugs. They gave the agent only one change per bug, where in production it could find the bug from a later one. The test used Gemini 3.6 flash, the same model as production BigSleep at that time.
In the other direction, fuzzers get a head start in production. Fuzzilli usually picks up new changes within hours. BigSleep runs once a day and reports crashes with a delay of one to two days.
The study also covers one target. V8 is a mature, heavily tested engine, and the researchers leave other targets and tools to future work. It is one careful data point, not a general ranking.
What Project Zero could not say
The report does not compare cost. Project Zero says effort went into making Clusterfuzz more efficient but not BigSleep. Neither system is tuned for cost per bug. A leader cannot yet read a price-performance verdict from this work.
Questions to put to your security team
Project Zero's advice to vendors is to pair fuzzing with agentic analysis of their own code, so that more flaws are reported before users feel them. Leaders can turn that into concrete questions.
First, does your team run both kinds of testing on code that faces customers? If it relies on one, which bugs would the other find?
Second, how are findings deduplicated? Two tools reporting the same flaw can look like twice the risk, and two reports of different flaws can look like one.
Third, how fast does each method see new code? A tool that checks changes within hours and one that checks daily cover different windows.
Fourth, what do you measure? Report counts alone mislead. Project Zero counted only bugs that received a fix.
Google's own finding is that the new tool and the old one see different bugs. Any programme should treat that as a reason to widen its coverage, not to swap one method for another.
Produced by the WebPulse Newsroom with AI assistance from the original reporting credited below, and checked against that source by our editorial review. How we use AI.
Original reporting: Google Project Zero.
