Re:Cloudflare adds option to block AI scrapers and crawlers
by altdataseller on Thursday June 27, 2024 @04:18PM
(#40814625)
How does Cloudflare know whether a bot is scraping for AI purposes or any other? Or are they blocking all bots but basically adding AI to the feature because AI is the cool thing right now?
[ Parent
| on HN ]
Re:Cloudflare adds option to block AI scrapers and crawlers
by rsynnott on Friday June 28, 2024 @05:38AM
(#40819060)
I assume a correct robots.txt, plus attempt to kill off robots.txt evaders; if you could do that reliably (the second bit is the tricky bit, of course), you could selectively exclude any bots you want.