Re:Comparing AI agents to cybersecurity professionals in rea...
by mens_rea on Tuesday January 20, 2026 @01:23AM
(#46688558)
Deeply flawed paper for several reasons:
* Small data set of 2 runs (!!)
* Exaggerated claims (saying A1 beat 50% of testers, yet only 4/10 testers found LESS vulns than A1, and A1 had a nearly 50% false positive rate)
* AI agents were given 16 hours while human testers were given 10
* Their human testers gave up when a modern browser refused to open a webpage with weak TLS ciphers so....clearly not professional testers unless the bar is REALLY low these days
* Small data set of 2 runs (!!)
* Exaggerated claims (saying A1 beat 50% of testers, yet only 4/10 testers found LESS vulns than A1, and A1 had a nearly 50% false positive rate)
* AI agents were given 16 hours while human testers were given 10
* Their human testers gave up when a modern browser refused to open a webpage with weak TLS ciphers so....clearly not professional testers unless the bar is REALLY low these days