News for hackers. Stuff that matters.

A comment on Cloudflare to introduce pay-per-crawl for AI bots. Parent comment

Re:Cloudflare to introduce pay-per-crawl for AI bots by bgwalter on Tuesday July 01, 2025 @07:55AM (#44432973)
If you don't want payment, there is:

https://anubis.techaro.lol/

Used by https://gcc.gnu.org/bugzilla/ for example. It is less annoying than CAPTCHA/Turnstile/whatever because the proof of work runs automatically.

[ Parent | on HN ]
Re:Cloudflare to introduce pay-per-crawl for AI bots by marginalia_nu on Tuesday July 01, 2025 @08:09AM (#44433070)
Sadly they seem to be getting through this one lately. Had a scraper hitting me with 80 qps punching straight through Anubis. Had to set up a global rate limit that browned out the functionality they were interested in[1] in excessive load.

[1] This form https://marginalia-search.com/site/news.ycombinator.com

[ Parent | on HN ]
Re:Cloudflare to introduce pay-per-crawl for AI bots by xena on Tuesday July 01, 2025 @12:10PM (#44435348)
Probably headless chrome. I'm going to investigate.
[ Parent | on HN ]
Re:Cloudflare to introduce pay-per-crawl for AI bots by marginalia_nu on Tuesday July 01, 2025 @03:49PM (#44437405)
They were hitting me at an incredible rate sustained load for like a week, seems like a lot of resources if it is indeed headless chrome.

Is there anything I can extract, fingerprint wise, if they come back to work out what's going on?

[ Parent | on HN ]
Re:Cloudflare to introduce pay-per-crawl for AI bots by gen6acd60af on Tuesday July 01, 2025 @08:09AM (#44433071)
See also (AFAIK most of these support JSless challenges out of the box): haproxy-protection, go-away, anticrawl
[ Parent | on HN ]
Re:Cloudflare to introduce pay-per-crawl for AI bots by xena on Tuesday July 01, 2025 @08:42AM (#44433352)
Anubis does too: https://anubis.techaro.lol/docs/admin/configuration/challeng...
[ Parent | on HN ]