News for hackers. Stuff that matters.
Re:Cloudflare's new AI traffic options for customers by dannyw on Sunday July 26, 2026 @09:05PM (#49064025)
The western AI crawlers are generally very-well behaved. GPTBot and ClaudeBot are nice crawlers and I see them respecting my robots.txt.

There are of course plenty of mystery, obfuscated/camouflaged scrapers/crawlers from god knows whom. Thankfully they are easy to spot and ban, although I've definitely thought about deliberating sending them poisoned data.

There are no comments on this comment yet.