Yeah, having a layer of human experts to sanity check and weed out hallucinated false positive issues seems like an important part of this process:> To ensure that Claude hadn’t hallucinated bugs (i.e., invented problems that don’t exist, a problem that increasingly is placing an undue burden on open source developers), we validated every bug extensively before reporting it. [...] for our initial round of findings, our own security researchers validated each vulnerability and wrote patches by hand. As the volume of findings grew, we brought in external (human) security researchers to help with validation and patch development.
Based on the experiences shared by curl's maintainers over the last couple of years, resulting in them ending their bug bounty program [1] [2] [3], I'd suggest the "growing risk of LLM-discovered [security issues]" is primarily maintainers being buried under a deluge of low-effort zero-value LLM-hallucinated false positive security issue reports, where the reporter copy-pastes LLM output without validation.
[1] https://daniel.haxx.se/blog/2026/02/03/open-source-security-...
[2] https://daniel.haxx.se/blog/2026/01/26/the-end-of-the-curl-b...
[3] https://daniel.haxx.se/blog/2025/07/14/death-by-a-thousand-s...
2/7/2026
at
6:24:31 AM
Ending a bug bounty program seems like a mistake.Why not just change the incentives? Don't pay for patches. Move the money over to human review of the infinite cesspool with an emphasis on how the findings are presented. Maintainers rank and filter by how concise the reviews are and how critical the bugs are. Stop allowing wide open pull requests for bugs and make that it's own new workflow.
Bugs rarely happen in isolation and many are regressions. Many are related to features added or refactors. Fixing bugs should be more about understanding the nature of the project than just playing whack-a-mole. LLMs don't have as good of a memory as humans and much of the meta discussion would be out-of-band for the LLMs. We shouldn't be paying for monkey work. We should be paying the humans that deeply understand "the lore" of the project and can apply it in a meaningful way.
In the first place, it's a long time coming that some maintainers feel the pressure to take the direction of the projects more seriously, and in some cases let others step up. So many open source projects need to be stop being the stereotype of lone genius pet projects or cultish power grabs. When people whine about open source not getting paid, this is the real reason why. It's not that the money or value isn't there, but a lack of confidence in the maintainers.
by sublinear