News for hackers. Stuff that matters.
Re:I was wrong about robots.txt by knorker on Thursday July 17, 2025 @04:49AM (#44591162)
The most annoying thing about being a good bot owner, in my experience, is when you get complaints about it misbehaving, only to find that it was actually somebody malicious who wrote their own abusive bot, but is using your bot's user agent.
Re:I was wrong about robots.txt by pimterry on Thursday July 17, 2025 @05:56AM (#44591539)
Cloudflare have some new bot verification proposals designed to fix this, with cryptographic proofs that the user-agent is who they say they are: https://blog.cloudflare.com/web-bot-auth/.
Re:I was wrong about robots.txt by EPendragon on Thursday July 17, 2025 @09:34AM (#44593227)
It is awesome to see that there are efforts to improve upon the old standards with modern web security standards