News for hackers. Stuff that matters.

A comment on AI bots everywhere. Does anyone have a good whitelist for robots.txt?.

Re:AI bots everywhere. Does anyone have a good whitelist for... by palmfacehn on Tuesday January 28, 2025 @11:11PM (#42861419)
Blocking Amazon, Huawei, China Telecom and few other ASNs did it for me. You should be able to cut, sort -u your log files to find the biggest offenders by user-agent if they truly obey robots.txt.
[ Parent | on HN ]

There are no comments on this comment yet.