News for hackers. Stuff that matters.
Re:Applebot, the web crawler for Apple by Aissen on Thursday May 07, 2015 @03:38AM (#9503743)
Putting anything on the internet exposes your "finite resources" to attackers. I'm sorry, but robots.txt is just a courtesy offered to you, and if you're doing serious business, you shouldn't rely on it. Use authentication, rate-limiting, captchas, DDoS-protecting CDNs, and enforce the limitations at the source, don't rely on nice people respecting your robots.txt.

There are no comments on this comment yet.