News for hackers. Stuff that matters.
Re:The Web Is Broken – Botnet Part 2 by jeroenhd on Sunday April 20, 2025 @03:54AM (#43742257)
The best solution I've seen is to hit everyone with a proof of work wall and whitelist the scrapers that are welcome (search engines and such).

Running SHA hash calculations for a second or so once every week is not bad for users, but with scrapers constantly starting new sessions they end up spending most of their time running useless Javascript, slowing the down significantly.

The most effective alternative to proof of work calculations seems to be remote attestation. The downside is that you're getting captchas if you're one of the 0.1% who disable secure boot and run Linux, but the vast majority of web users will live a captcha free life. This same mechanism could in theory also be used to authenticate welcome scrapers rather than relying on pure IP whitelists.

Re:The Web Is Broken – Botnet Part 2 by ognarb on Sunday April 27, 2025 @04:09PM (#43814718)
The issue is that it would require normal user to also do the same, which is suboptimal from a privacy point of view.