Re:Google competitor DuckDuckGo says it's getting shut out
by Matt_Cutts on Thursday November 22, 2012 @11:37AM
(#4819267)
That's interesting.
Gabriel, does DuckDuckGo's crawler have a distinct user agent? Can you talk more about how DuckDuckGo observes/respects robots.txt?
Interestingly, the fetches do not have a user-agent that identifies itself as the DDG crawler:
Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; .NET CLR 1.1.4322)
I'm assuming this is the crawler because it does not fetch anything besides text/html.