Re:I rebuilt my blog's cache. Bots are the audience now
by tardedmeme on Sunday May 03, 2026 @01:30PM
(#47999297)
They use different user-agent strings. The crawlers obfuscate themselves and use residential proxies. The agents call themselves ChatGPT-User. Of course Cloudflare wants OpenAI to pay them for not blocking ChatGPT-User by default.
[ Parent
| on HN ]
Re:I rebuilt my blog's cache. Bots are the audience now
by faangguyindia on Sunday May 03, 2026 @01:36PM
(#47999370)
It's true, crawlers used for AI training don't say they are crawlers at all.