Re:Google: Bing Is Cheating, Copying Our Search Results
by poacher69 on Tuesday February 01, 2011 @10:47AM
(#2165875)
are you claiming that google never scrapes bing search results pages? or any other search result pages?
[ Parent
| on HN ]
Re:Google: Bing Is Cheating, Copying Our Search Results
by Matt_Cutts on Tuesday February 01, 2011 @10:55AM
(#2165902)
poacher69, we crawl the public web. Anyone that blocks us out with robots.txt, we won't crawl. If you check bing.com/robots.txt, it has "Disallow: /search" . So no, we won't crawl Bing's search results pages. If anything, users tend to complain when search results from Lycos or wherever show up in Google.
Re:Google: Bing Is Cheating, Copying Our Search Results
by underdown on Tuesday February 01, 2011 @03:23PM
(#2167187)
http://www.bing.com/robots.txt
User-agent: *
Disallow: /search
Funny thing: http://www.google.com/search?q=site%3Abing.com%2Fsearch%2F
I was gonna call out Matt for crawling bing's search results but I'm guessing Microsoft hasn't realized they return results from the /Search/ folder. ;)
Re:Google: Bing Is Cheating, Copying Our Search Results
by nitrogen on Tuesday February 01, 2011 @04:50PM
(#2167577)
Once again Microsoft is bitten by expecting case insensitivity.
Re:Google: Bing Is Cheating, Copying Our Search Results
by poacher69 on Tuesday February 01, 2011 @10:00PM
(#2168736)
matt, how does google do competitive relevance evaluations without scraping Bing?
Re:Google: Bing Is Cheating, Copying Our Search Results
by illdave on Tuesday February 01, 2011 @10:52AM
(#2165893)
From my experience, Googlebot doesn't crawl pages that are blocked in robots.txt files. Check out Bing's robots.txt: http://bing.com/robots.txt - notice how /search is disallowed. That typically means that Googlebot isn't able to access that page. The same for the other search engines, it's more down to if they specify (through robots.txt) that Googlebot isn't allowed to crawl those results.