News for hackers. Stuff that matters.
Re:Applebot, the web crawler for Apple by d0ugie on Wednesday May 06, 2015 @03:35PM (#9501269)
Note that serving bots, especially for media-rich sites, eats heavily into a finite resource.

For those of us whose resources are especially tight, blocking Yandex, Baidu and MSN may be very helpful, your ideals notwithstanding.

Re:Applebot, the web crawler for Apple by Aissen on Thursday May 07, 2015 @03:38AM (#9503743)
Putting anything on the internet exposes your "finite resources" to attackers. I'm sorry, but robots.txt is just a courtesy offered to you, and if you're doing serious business, you shouldn't rely on it. Use authentication, rate-limiting, captchas, DDoS-protecting CDNs, and enforce the limitations at the source, don't rely on nice people respecting your robots.txt.