News for hackers. Stuff that matters.

A comment on AI bots everywhere. Does anyone have a good whitelist for robots.txt?. Parent comment

Re:AI bots everywhere. Does anyone have a good whitelist for... by userbinator on Wednesday January 29, 2025 @12:32AM (#42861897)
More precisely described as "block non-sanctioned user agents".

Using that feature will ensure I never visit your site again.

[ Parent | on HN ]
Re:AI bots everywhere. Does anyone have a good whitelist for... by aetherspawn on Wednesday January 29, 2025 @12:50AM (#42861973)
It's a free wiki so not visiting is a right you have, sure .. but also OP shouldn't have to spend loads of $ to host a free wiki and at the same time support a very bespoke thing such as non standard user agents.
[ Parent | on HN ]
Re:AI bots everywhere. Does anyone have a good whitelist for... by zaphodias on Wednesday January 29, 2025 @10:02AM (#42865652)
> is a right

I believe they're saying that cloudflare will block them just for using a blacklisted client, even if they're legit users and not bots

[ Parent | on HN ]
Re:AI bots everywhere. Does anyone have a good whitelist for... by userbinator on Friday January 31, 2025 @12:48AM (#42885010)
Not even that; it's a whitelist, not a blacklist, and the only clients whitelisted are essentially those of Big Browser.
[ Parent | on HN ]