News for hackers. Stuff that matters.

A comment on I was wrong about robots.txt. Parent comment

Re:I was wrong about robots.txt by sltkr on Wednesday July 16, 2025 @11:42PM (#44589475)
The really bad actors are going to ignore robots.txt entirely. You might as well be nice to the crawlers that respect robots.txt.
[ Parent | on HN ]
Re:I was wrong about robots.txt by PeterStuer on Thursday July 17, 2025 @01:00AM (#44589838)
Even if you want to play nice, robots.txt is a catch-22, as accessing it is taken as a signal you are a 'bot' by malconfigured anti-bot 'solutions'.
[ Parent | on HN ]