News for hackers. Stuff that matters.

A comment on AI bots everywhere. Does anyone have a good whitelist for robots.txt?. Parent comment

Re:AI bots everywhere. Does anyone have a good whitelist for... by CommanderData on Wednesday January 29, 2025 @02:44AM (#42862574)
Robots.txt? Just send a fake user agent.
[ Parent | on HN ]

There are no comments on this comment yet.