AI bots everywhere. Does anyone have a good whitelist for robots.txt?

Re:AI bots everywhere. Does anyone have a good whitelist for... by gigaArpit on Wednesday January 29, 2025 @12:01AM (#42861738)

Robots.txt is a joke, use Cloudflare's Block AI Bots feature if you are using Cloudflare.

[ Parent | on HN ]

Re:AI bots everywhere. Does anyone have a good whitelist for... by graeme on Wednesday January 29, 2025 @12:12AM (#42861796)

On this point, if you turn on bot fight mode it also says it blocks verified bots.

But, bot fight mode says "there is a newer version of this setting" however it does not link to it.

Anyone have any insight on the blocked verified bots or the supposed new version?

[ Parent | on HN ]

Re:AI bots everywhere. Does anyone have a good whitelist for... by userbinator on Wednesday January 29, 2025 @12:32AM (#42861897)

More precisely described as "block non-sanctioned user agents".

Using that feature will ensure I never visit your site again.

[ Parent | on HN ]

Re:AI bots everywhere. Does anyone have a good whitelist for... by aetherspawn on Wednesday January 29, 2025 @12:50AM (#42861973)

It's a free wiki so not visiting is a right you have, sure .. but also OP shouldn't have to spend loads of $ to host a free wiki and at the same time support a very bespoke thing such as non standard user agents.

[ Parent | on HN ]

Re:AI bots everywhere. Does anyone have a good whitelist for... by zaphodias on Wednesday January 29, 2025 @10:02AM (#42865652)

> is a right

I believe they're saying that cloudflare will block them just for using a blacklisted client, even if they're legit users and not bots

[ Parent | on HN ]

Re:AI bots everywhere. Does anyone have a good whitelist for... by userbinator on Friday January 31, 2025 @12:48AM (#42885010)

Not even that; it's a whitelist, not a blacklist, and the only clients whitelisted are essentially those of Big Browser.

[ Parent | on HN ]