Re:AI bots everywhere. Does anyone have a good whitelist for...
by scoofy on Tuesday January 28, 2025 @11:28PM
(#42861540)
Will this still allow Google and Bing and Archive bots?
[ Parent
| on HN ]
Re:AI bots everywhere. Does anyone have a good whitelist for...
by aetherspawn on Tuesday January 28, 2025 @11:48PM
(#42861658)
I quickly checked for you, and my Cloudflare account currently has the following option: "Block AI Bots" under xxx.com > Security > Bots which I think does what you want.
I am not paying for Cloudflare and it allows me to enable this option. If you use the (other) Bot Fight Mode or captcha options, then yes, you will block crawlers. AFAIK specific bot user agents can also be blocked (or allowed) using a Page Rule, which is also a free tier feature.