News for hackers. Stuff that matters.
Re:AI Bots Are Now a Significant Source of Web Traffic by hnrayst on Wednesday February 04, 2026 @10:43AM (#46887211)
This looks like the new robots.txt arms race. Cloudflare recently said sites blocking GPTBot/ClaudeBot outnumber Googlebot blocks by a lot, and more publishers are turning on AI crawl controls. Curious how much of the “bot traffic” is known crawlers vs disguised scraping.

There are no comments on this comment yet.