News for hackers. Stuff that matters.

A comment on Botspam apocalypse. Parent comment

Re:Botspam apocalypse by BasiliusCarver on Thursday August 04, 2022 @03:57AM (#32340365)
One of the things I’ve done before among the other suggestions is to put a hidden link like /followmeifyouscraping.html in the landing page to get a bit of info about scraping volume and then you can use fail2ban filters to block if it’s visited if you want
[ Parent | on HN ]
Re:Botspam apocalypse by jiggawatts on Thursday August 04, 2022 @04:13AM (#32340471)
I would add that link to robots.txt as an exclusion.

That way well-behaved search engines won’t be affected, but naive scrapers get auto-banned.

[ Parent | on HN ]