Re:Google: Bing Is Cheating, Copying Our Search Results
by Matt_Cutts on Tuesday February 01, 2011 @10:55AM
(#2165902)
poacher69, we crawl the public web. Anyone that blocks us out with robots.txt, we won't crawl. If you check bing.com/robots.txt, it has "Disallow: /search" . So no, we won't crawl Bing's search results pages. If anything, users tend to complain when search results from Lycos or wherever show up in Google.
[ Parent
| on HN ]
Re:Google: Bing Is Cheating, Copying Our Search Results
by underdown on Tuesday February 01, 2011 @03:23PM
(#2167187)
http://www.bing.com/robots.txt
User-agent: *
Disallow: /search
Funny thing: http://www.google.com/search?q=site%3Abing.com%2Fsearch%2F
I was gonna call out Matt for crawling bing's search results but I'm guessing Microsoft hasn't realized they return results from the /Search/ folder. ;)