News for hackers. Stuff that matters.
Re:Applebot, the web crawler for Apple by frankacter on Wednesday May 06, 2015 @05:30AM (#9497447)
>If robots instructions don't mention Applebot but do mention Googlebot, the Apple robot will follow Googlebot instructions.

So if I set in my robots.txt to disallow all bots except Googlebot, Applebot will index anyway? I don't think I like that precedent.

Re:Applebot, the web crawler for Apple by Aissen on Wednesday May 06, 2015 @06:56AM (#9497711)
I've said it before, and I'll say it again: blocking all but certain bots is the best way to block innovation. It's bad for you, it's bad for the ecosystem. Bad for the ecosystem because incumbents that want to respect robots.txt to propose new services will have an unfair disadvantage. It's bad for you because it'll just give more power to Google et al regarding your incoming traffic (and you'll have to follow their rules, like every SEO is doing right now).
Re:Applebot, the web crawler for Apple by frankacter on Thursday May 07, 2015 @12:14AM (#9503251)
But that is a choice that a website operator has the authority to make given current standards. Apple is ignoring the wishes of a website operator and piggybacking on the trust that a website operator has place in GoogleBot.
Re:Applebot, the web crawler for Apple by d0ugie on Wednesday May 06, 2015 @03:35PM (#9501269)
Note that serving bots, especially for media-rich sites, eats heavily into a finite resource.

For those of us whose resources are especially tight, blocking Yandex, Baidu and MSN may be very helpful, your ideals notwithstanding.

Re:Applebot, the web crawler for Apple by Aissen on Thursday May 07, 2015 @03:38AM (#9503743)
Putting anything on the internet exposes your "finite resources" to attackers. I'm sorry, but robots.txt is just a courtesy offered to you, and if you're doing serious business, you shouldn't rely on it. Use authentication, rate-limiting, captchas, DDoS-protecting CDNs, and enforce the limitations at the source, don't rely on nice people respecting your robots.txt.
Re:Applebot, the web crawler for Apple by WorldWideWayne on Wednesday May 06, 2015 @11:52AM (#9499550)
I wonder what are the legalities of this type of discrimination? Retail businesses aren't allowed to arbitrarily refuse service, they have to follow certain rules and be consistent.
Re:Applebot, the web crawler for Apple by MCRed on Wednesday May 06, 2015 @07:00PM (#9502321)
Is there legal backing to enforce obeying robots.txt, or is it a guideline?
Re:Applebot, the web crawler for Apple by WorldWideWayne on Friday May 08, 2015 @10:56AM (#9512029)
There are some court cases that involve robots.txt but I'm not sure because there was a lot of reading to do and I was hoping someone who knew would just come along and summarize it for us :)
Re:Applebot, the web crawler for Apple by jwr on Wednesday May 06, 2015 @10:02AM (#9498652)
I actually loved this part of the page being discussed.

The whole idea of versioning content based on who accesses it is broken and fundamentally at odds with the idea of the open web. Same goes for user-agent string madness, by the way. Yes, we should be able to tell robots from humans, but otherwise, it's supposed to be the Web.

Incidentally, this hits close to a pain point: I find it extremely annoying when publishers (like Elsevier) hide content behind a paywall, but still expose it to Googlebot for indexing. The result is that you are able to find a scientific article, which is not accessible (but Googlebot has cached snippets). This goes against Google's own guidelines (they used to tell people that Googlebot must not see different content from browsers). And it goes against the whole idea of the Web: if you want to hide stuff behind a paywall, do so — but then it is no longer accessible.

Going back to Applebot, I love the fact that they will now follow Googlebot instructions. Hopefully people will stop distinguishing who accesses content.

Re:Applebot, the web crawler for Apple by soylentcola on Wednesday May 06, 2015 @10:28AM (#9498849)
I remember a big issue with user-agent filtering when Google was still trying to make Google TV work. At the time, loads of TV networks had free, full streaming episodes of shows on their websites as a way to capture some ad revenue that would be lost if people were torrenting or whatever in order to see shows they missed. Google TV was attempting to list those episodes alongside whatever was currently live on TV via your cable/sat/antenna and stuff from other sites like Youtube.

The idea was that instead of having to go to all sorts of places to find content, it would show you what was available at a given time based on what you were looking for.

...and then all of the network sites and the free Hulu stuff got put behind a user agent filter and essentially wiped out a huge part of GTV's reason for existence. The goal was to bring all of the free content into one place but the networks didn't want you watching a free stream in lieu of a cable broadcast. They wanted you to watch cable on your living room TV and only use the free streaming episodes from the computer in your office as a backup.

Same goes for services where web viewing/listening is free but if you try to access it from a mobile web browser, you have to either fool the site or subscribe to some mobile version.

Re:Applebot, the web crawler for Apple by gutnor on Wednesday May 06, 2015 @06:05AM (#9497569)
That's maybe jumping to conclusion.

I interpreted the sentence less literally and more like "in absence of rule, default to GoogleBot ones"

So if you put a wildcard rule forbidding access and a specific one allowing access to GoogleBot, AppleBot will honour the wildcard one.

That's how I would have coded it anyway: parse the rule for current agent string, if no rule applies, run it again with GoogleBot one before assuming that website does not contain restrictions.

Re:Applebot, the web crawler for Apple by koyote on Wednesday May 06, 2015 @06:50AM (#9497697)
"If robots instructions don't mention Applebot"

I would assume this means that it will follow GoogleBot unless you specifically mention AppleBot by name and not by using a wildcard.

So a User-agent: * would be ignored if a User-agent: GoogleBot is found.

Re:Applebot, the web crawler for Apple by Bulk70 on Wednesday May 06, 2015 @05:34AM (#9497464)
Serious question, because I can't imagine your use case - under what circumstances would you wan't to block all bots except one?
Re:Applebot, the web crawler for Apple by nemothekid on Wednesday May 06, 2015 @06:40AM (#9497665)
Facebook blocks all bots except a select few to prevent site scraping - https://www.facebook.com/robots.txt
Re:Applebot, the web crawler for Apple by mbel on Wednesday May 06, 2015 @05:42AM (#9497496)
I'm not an expert but my guess is: limiting bot traffic, but keeping the site available for the most popular search engine.
Re:Applebot, the web crawler for Apple by mrweasel on Wednesday May 06, 2015 @06:45AM (#9497680)
My experience is that the worst bots don't respect robots.txt anyway.

Getting crawled by the major search engines typically isn't that bad, they tend to know what they're doing. Getting hammered by some crappy local search engine is what's annoying.

We don't limit any bots, except once where we completely blocked Eniro in our firewall. Google, Bing and a ton of other could index at the same time, with no issue. Eniro for some reason decided to just index way to much at once, no reaction to robots.txt and no reply from the email they so kindly included in the headers.

But I see your point, it's just a bit sad when Google has become "The Internet".

Re:Applebot, the web crawler for Apple by karmakaze on Wednesday May 06, 2015 @02:08PM (#9500611)
I thought FB was the internet. Googlebot is just the Kleenex of indexers.
Re:Applebot, the web crawler for Apple by gojomo on Wednesday May 06, 2015 @06:02AM (#9497557)
Owner of GOOG stock, maybe?
Re:Applebot, the web crawler for Apple by RexRollman on Wednesday May 06, 2015 @05:42AM (#9497492)
Maybe for ethical reasons or some kind of exclusive agreement with one particular search vendor?
Re:Applebot, the web crawler for Apple by bad_user on Wednesday May 06, 2015 @06:32AM (#9497639)
Is it of any importance?
Re:Applebot, the web crawler for Apple by mobiplayer on Wednesday May 06, 2015 @09:41AM (#9498476)
That's what you get when many people code with only one company in mind. It happened in the past and it is happening again, whether we like it or not. Even Microsoft was kinda masquerading early versions of Edge as Chrome.
Re:Applebot, the web crawler for Apple by protomyth on Wednesday May 06, 2015 @09:55AM (#9498599)
"Even Microsoft was kinda masquerading early versions of Edge as Chrome"

I would assume that was more to keep the tech press from seeing it.

Re:Applebot, the web crawler for Apple by mobiplayer on Thursday May 07, 2015 @04:23PM (#9508013)
I don't know about the not public betas, but as soon as there was a public Windows 10 TP and you could swap IE's engine to Edge I'm pretty sure they had "Edge" at the end of the user-agent, but they had the rest of the Chrome UA so most of the websites out there considered it as being Chrome.

I may also be completely wrong :)

Re:Applebot, the web crawler for Apple by karmakaze on Wednesday May 06, 2015 @11:50AM (#9499526)
Why? Applebot is attempting to adhere to the spirit of robots.txt rather than the letter. If the site owner cares otherwise, don't just explicitly name one bot. Is `user-agent: *bot` valid?
Re:Applebot, the web crawler for Apple by Tloewald on Wednesday May 06, 2015 @09:10AM (#9498299)
You'd prefer they didn't honor robots.txt at all?