Cloudflare to introduce pay-per-crawl for AI bots

Re:Cloudflare to introduce pay-per-crawl for AI bots by suyash on Tuesday July 01, 2025 @06:46AM (#44432533)

Nice to see someone addressing this annoying problem, I'm seeing first hand bot traffic go up as they are just gobbling up data. However instead of relying on Cloudflare, it would be better to have a open source protocol that handles permission and payment for crawlers/scraper.

[ Parent | on HN ]

Re:Cloudflare to introduce pay-per-crawl for AI bots by bgwalter on Tuesday July 01, 2025 @07:55AM (#44432973)

If you don't want payment, there is:

https://anubis.techaro.lol/

Used by https://gcc.gnu.org/bugzilla/ for example. It is less annoying than CAPTCHA/Turnstile/whatever because the proof of work runs automatically.

[ Parent | on HN ]

Re:Cloudflare to introduce pay-per-crawl for AI bots by marginalia_nu on Tuesday July 01, 2025 @08:09AM (#44433070)

Sadly they seem to be getting through this one lately. Had a scraper hitting me with 80 qps punching straight through Anubis. Had to set up a global rate limit that browned out the functionality they were interested in[1] in excessive load.

[1] This form https://marginalia-search.com/site/news.ycombinator.com

[ Parent | on HN ]

Re:Cloudflare to introduce pay-per-crawl for AI bots by xena on Tuesday July 01, 2025 @12:10PM (#44435348)

Probably headless chrome. I'm going to investigate.

[ Parent | on HN ]

Re:Cloudflare to introduce pay-per-crawl for AI bots by marginalia_nu on Tuesday July 01, 2025 @03:49PM (#44437405)

They were hitting me at an incredible rate sustained load for like a week, seems like a lot of resources if it is indeed headless chrome.

Is there anything I can extract, fingerprint wise, if they come back to work out what's going on?

[ Parent | on HN ]

Re:Cloudflare to introduce pay-per-crawl for AI bots by gen6acd60af on Tuesday July 01, 2025 @08:09AM (#44433071)

See also (AFAIK most of these support JSless challenges out of the box): haproxy-protection, go-away, anticrawl

[ Parent | on HN ]

Re:Cloudflare to introduce pay-per-crawl for AI bots by xena on Tuesday July 01, 2025 @08:42AM (#44433352)

Anubis does too: https://anubis.techaro.lol/docs/admin/configuration/challeng...

[ Parent | on HN ]

Re:Cloudflare to introduce pay-per-crawl for AI bots by rswail on Tuesday July 01, 2025 @06:59AM (#44432610)

The protocol that Cloudflare are proposing could be implemented by anyone. There will need to be ways for crawlers to register and pay.

CF is acting as the merchant of record, so they will be the ones billing, it's unclear what cut of the price they will take (if any) or if they will include it in their bundled services.

This should be expanded to allow for:

* micropayments and subscriptions

* integration with the browser UI/UX

* multiple currencies

* implementation of multiple payment systems, including national instant settlement systems like UPI, NPP, FedNow etc.

[ Parent | on HN ]

Re:Cloudflare to introduce pay-per-crawl for AI bots by Leynos on Tuesday July 01, 2025 @07:22AM (#44432746)

Is this companies collecting data for model training, or is it agentic tools operating on behalf of users?

[ Parent | on HN ]

Re:Cloudflare to introduce pay-per-crawl for AI bots by xena on Tuesday July 01, 2025 @08:04AM (#44433037)

Whatever it is, I've seen the commons abuse Gitlab servers so hard they peg 64 high wattage server cores 24/7. Installing mitigations cut their power bill in half.

[ Parent | on HN ]

Re:Cloudflare to introduce pay-per-crawl for AI bots by Melonai on Tuesday July 01, 2025 @08:20AM (#44433157)

I think in the grand scheme of things barely anyone uses agents (as of now) to crawl sites quickly apart from maybe a quick Google search or two. At least that's been my observation of my non-technical field friends using LLMs.

From what it looks like in the web logs it is in fact the same few AI company web crawlers constantly crawling and recrawling the same URLs over and over, presumably to get even the slightest advantage over each other, as they are definitely in the zero-sum mindset currently.

[ Parent | on HN ]