News for hackers. Stuff that matters.
Re:Applebot, the web crawler for Apple by Aissen on Wednesday May 06, 2015 @06:56AM (#9497711)
I've said it before, and I'll say it again: blocking all but certain bots is the best way to block innovation. It's bad for you, it's bad for the ecosystem. Bad for the ecosystem because incumbents that want to respect robots.txt to propose new services will have an unfair disadvantage. It's bad for you because it'll just give more power to Google et al regarding your incoming traffic (and you'll have to follow their rules, like every SEO is doing right now).
Re:Applebot, the web crawler for Apple by frankacter on Thursday May 07, 2015 @12:14AM (#9503251)
But that is a choice that a website operator has the authority to make given current standards. Apple is ignoring the wishes of a website operator and piggybacking on the trust that a website operator has place in GoogleBot.
Re:Applebot, the web crawler for Apple by d0ugie on Wednesday May 06, 2015 @03:35PM (#9501269)
Note that serving bots, especially for media-rich sites, eats heavily into a finite resource.

For those of us whose resources are especially tight, blocking Yandex, Baidu and MSN may be very helpful, your ideals notwithstanding.

Re:Applebot, the web crawler for Apple by Aissen on Thursday May 07, 2015 @03:38AM (#9503743)
Putting anything on the internet exposes your "finite resources" to attackers. I'm sorry, but robots.txt is just a courtesy offered to you, and if you're doing serious business, you shouldn't rely on it. Use authentication, rate-limiting, captchas, DDoS-protecting CDNs, and enforce the limitations at the source, don't rely on nice people respecting your robots.txt.
Re:Applebot, the web crawler for Apple by WorldWideWayne on Wednesday May 06, 2015 @11:52AM (#9499550)
I wonder what are the legalities of this type of discrimination? Retail businesses aren't allowed to arbitrarily refuse service, they have to follow certain rules and be consistent.
Re:Applebot, the web crawler for Apple by MCRed on Wednesday May 06, 2015 @07:00PM (#9502321)
Is there legal backing to enforce obeying robots.txt, or is it a guideline?
Re:Applebot, the web crawler for Apple by WorldWideWayne on Friday May 08, 2015 @10:56AM (#9512029)
There are some court cases that involve robots.txt but I'm not sure because there was a lot of reading to do and I was hoping someone who knew would just come along and summarize it for us :)