AI bots everywhere. Does anyone have a good whitelist for robots.txt?

Re:AI bots everywhere. Does anyone have a good whitelist for... by pdntspa on Wednesday January 29, 2025 @01:07AM (#42862061)

It's so cute how people think anyone obeys robots.txt :)

[ Parent | on HN ]

Re:AI bots everywhere. Does anyone have a good whitelist for... by memhole on Wednesday January 29, 2025 @02:28AM (#42862498)

I personally don’t even bother with it for these reasons.

[ Parent | on HN ]

Re:AI bots everywhere. Does anyone have a good whitelist for... by CommanderData on Wednesday January 29, 2025 @02:44AM (#42862574)

Robots.txt? Just send a fake user agent.

[ Parent | on HN ]

Re:AI bots everywhere. Does anyone have a good whitelist for... by skybrian on Wednesday January 29, 2025 @09:58PM (#42874312)

And why should anyone believe you when you didn’t give any evidence of anything?

[ Parent | on HN ]

Re:AI bots everywhere. Does anyone have a good whitelist for... by qup on Friday January 31, 2025 @09:53AM (#42888250)

He didn't even make a claim

[ Parent | on HN ]