Re:Applebot, the web crawler for Apple
by Aissen on Thursday May 07, 2015 @03:38AM
(#9503743)
Putting anything on the internet exposes your "finite resources" to attackers. I'm sorry, but robots.txt is just a courtesy offered to you, and if you're doing serious business, you shouldn't rely on it. Use authentication, rate-limiting, captchas, DDoS-protecting CDNs, and enforce the limitations at the source, don't rely on nice people respecting your robots.txt.
[ Parent
| on HN ]
There are no comments on this comment yet.