Re:AI Bots Are Now a Significant Source of Web Traffic
by hnrayst on Wednesday February 04, 2026 @10:43AM
(#46887211)
This looks like the new robots.txt arms race. Cloudflare recently said sites blocking GPTBot/ClaudeBot outnumber Googlebot blocks by a lot, and more publishers are turning on AI crawl controls. Curious how much of the “bot traffic” is known crawlers vs disguised scraping.
[ Parent
| on HN ]
There are no comments on this comment yet.