Re:Cloudflare's new AI traffic options for customers
by mysterydip on Sunday July 26, 2026 @09:48AM
(#49058228)
AI crawlers, famous for respecting robots.txt ;)
[ Parent
| on HN ]
Re:Cloudflare's new AI traffic options for customers
by dannyw on Sunday July 26, 2026 @09:05PM
(#49064025)
The western AI crawlers are generally very-well behaved. GPTBot and ClaudeBot are nice crawlers and I see them respecting my robots.txt.
There are of course plenty of mystery, obfuscated/camouflaged scrapers/crawlers from god knows whom. Thankfully they are easy to spot and ban, although I've definitely thought about deliberating sending them poisoned data.