News for hackers. Stuff that matters.

A comment on I was wrong about robots.txt. Parent comment

Re:I was wrong about robots.txt by PeterStuer on Thursday July 17, 2025 @01:00AM (#44589838)
Even if you want to play nice, robots.txt is a catch-22, as accessing it is taken as a signal you are a 'bot' by malconfigured anti-bot 'solutions'.
[ Parent | on HN ]

There are no comments on this comment yet.