Re:I was wrong about robots.txt
by hinkley on Thursday July 17, 2025 @01:23PM
(#44595737)
All our customers were in North American but we let a semi naughty bot from the UK scan us and I will never understand why. It was still sending us malformed URLs we purged from the site years ago. WTF.
[ Parent
| on HN ]
There are no comments on this comment yet.