News for hackers. Stuff that matters.

A comment on Cloudflare to introduce pay-per-crawl for AI bots. Parent comment

Re:Cloudflare to introduce pay-per-crawl for AI bots by marginalia_nu on Tuesday July 01, 2025 @08:09AM (#44433070)
Sadly they seem to be getting through this one lately. Had a scraper hitting me with 80 qps punching straight through Anubis. Had to set up a global rate limit that browned out the functionality they were interested in[1] in excessive load.

[1] This form https://marginalia-search.com/site/news.ycombinator.com

[ Parent | on HN ]
Re:Cloudflare to introduce pay-per-crawl for AI bots by xena on Tuesday July 01, 2025 @12:10PM (#44435348)
Probably headless chrome. I'm going to investigate.
[ Parent | on HN ]
Re:Cloudflare to introduce pay-per-crawl for AI bots by marginalia_nu on Tuesday July 01, 2025 @03:49PM (#44437405)
They were hitting me at an incredible rate sustained load for like a week, seems like a lot of resources if it is indeed headless chrome.

Is there anything I can extract, fingerprint wise, if they come back to work out what's going on?

[ Parent | on HN ]