News for hackers. Stuff that matters.
Re:Google competitor DuckDuckGo says it's getting shut out by metalruler on Thursday November 22, 2012 @07:07AM (#4818464)
I checked my logs and there are several fetches from 72.94.249.37 and 72.94.249.38, over a number of domains that I host. None are particularly popular as far as the greater internet is concerned; one is a semi private site that I set up for my daughter's photos, another is one that has not yet been developed, apart from a few words of text and an image.

Interestingly, the fetches do not have a user-agent that identifies itself as the DDG crawler:

Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; .NET CLR 1.1.4322)

I'm assuming this is the crawler because it does not fetch anything besides text/html.

Re:Google competitor DuckDuckGo says it's getting shut out by Matt_Cutts on Thursday November 22, 2012 @11:37AM (#4819267)
That's interesting.

Gabriel, does DuckDuckGo's crawler have a distinct user agent? Can you talk more about how DuckDuckGo observes/respects robots.txt?