LLM bug-finder and fuzzing find similar V8 bugs but only ~25% overlap
The claim
The authors assert that LLM-based bug finding and coverage-guided fuzzing discover largely distinct sets of V8 vulnerabilities, with only about a quarter overlapping.
Published as: “Fuzzers vs. AI: Comparing Fuzzilli and BigSleep on JavaScript Engines”
So what
LLM agents can substantially augment traditional fuzzing, increasing overall vulnerability coverage in complex software like V8.
The five-minute version
Project Zero compared the production outputs of Fuzzilli, a JavaScript engine fuzzer, and BigSleep, an LLM-driven bug-finding agent, on Chrome's V8 engine over six months. In that period the two systems reported a similar number of actionable vulnerabilities (45 vs 37) but only about 25% of those reports overlapped. A dedicated experiment attempting to rediscover historic Fuzzilli bugs with BigSleep achieved roughly 62% success, suggesting a higher theoretical overlap of around 60%. The authors conclude that the approaches are complementary, with LLM agents excelling at complex logic bugs and fuzzers still valuable for other classes of issues.
What they found 3
- Actionable vulnerability reports overlap is roughly 25% (20 of 81).
“we see an overlap of roughly 25% (20 out of a total of 81), with a slight majority of these found first by Fuzzilli and later by BigSleep.”
- In the dedicated experiment BigSleep rediscovered about 62% of historic Fuzzilli bugs.
“BigSleep was able to re-discover ~62% of the original bugs, while finding a different (but by now fixed) bug for another ~12% of commits.”
- Theoretical overlap estimated from dedicated experiments is roughly 60%.
“These experiments estimated overlap at roughly 60%.”
What undercuts it the paper’s own results
- The experiment only considered bugs that reproduce on a standard x64 d8 build without special flags, which may under-represent the full set of Fuzzilli findings.
- The production overlap may be higher than 25% because the methodology tends to underestimate duplicate reporting.
Stated limits 4
- Our experiment only looked at a subset of Fuzzilli bugs that reproduce on a standard x64 d8 fuzzer build and without special flags.
- Unfortunately, it’s challenging to make a comparison at this stage in the research.
- There’s a limited time window for overlaps: once a bug reported by either system is fixed, the other system no longer has a chance to rediscover it (as the bug won’t reproduce on Clusterfuzz and will therefore be discarded).
- For the model used in this experiment (Gemini 3.6 flash), the knowledge cutoff date is March 2026.
What’s new here 4
- First large-scale production comparison of a generative fuzzer versus an LLM-based bug-finding agent on V8.
- Measured real-world overlap of actionable vulnerability reports at ~25% and experimental overlap at ~60%.
- Showed that bug difficulty correlates for both methods, with hard-to-find bugs being hard for both.
- Identified specific bug classes where LLM agents have an advantage (complex logical reasoning, magic values) and where fuzzers excel (non-local bugs).
Artifacts & methods 5
- Fuzzillisystem
- BigSleepsystem
- Gemini 3.6 flashprimitive
- Chrome V8 JavaScript enginesystem
- Clusterfuzzsystem