News for hackers. Stuff that matters.

A comment on Cloudflare adds option to block AI scrapers and crawlers.

Re:Cloudflare adds option to block AI scrapers and crawlers by chollida1 on Thursday June 27, 2024 @04:17PM (#40814616)
what is the difference between a scraper or crawler and an "AI scraper/crawler"?

And how does cloudflare differentiate between them?

[ Parent | on HN ]
Re:Cloudflare adds option to block AI scrapers and crawlers by twelve40 on Thursday June 27, 2024 @05:09PM (#40815129)
it's explained pretty well in their blog post in another thread. They have a whitelist of bots catergorized by type, and ban everything else. How easy it is to spoof (just the UA, or UA+IP whatever) is not mentioned.
[ Parent | on HN ]
Re:Cloudflare adds option to block AI scrapers and crawlers by epanchin on Thursday June 27, 2024 @04:29PM (#40814743)
The latter doesn’t request robots.txt
[ Parent | on HN ]