News for hackers. Stuff that matters.
Re:Google competitor DuckDuckGo says it's getting shut out by joenathan on Thursday November 22, 2012 @12:22AM (#4817594)
According to DuckDuckGo's FAQ you are wrong http://help.duckduckgo.com/customer/portal/articles/216399-s...

The DuckDuckBot crawls and indexes the web. http://duckduckgo.com/duckduckbot.html

Re:Google competitor DuckDuckGo says it's getting shut out by ck2 on Thursday November 22, 2012 @12:35AM (#4817623)
I suspect it's trivial at best.

Every test search I've ever done on DDG shows near identical results to Bing.

I'd like to see a search that uses it's own data, examples?

Gigablast was the last serious third-party backend that had a chance for independent data. It's like old-school Google.

Gabriel should try to buy Gigablast and merge it with DDG so he has his own independent dataset.

Re:Google competitor DuckDuckGo says it's getting shut out by lubujackson on Thursday November 22, 2012 @01:33AM (#4817761)
Blekko has millions behind it and does a full index of the web.
Re:Google competitor DuckDuckGo says it's getting shut out by ck2 on Thursday November 22, 2012 @01:44AM (#4817795)
Ah now that is an interesting engine.

Checking to see if it's hit any of our sites.

3k pages, not bad. Data is kinda stale though.

Re:Google competitor DuckDuckGo says it's getting shut out by boyter on Thursday November 22, 2012 @04:13AM (#4818081)
Gigablasts creator has made this now http://procog.com/ the results are better then gigablast.
Re:Google competitor DuckDuckGo says it's getting shut out by boyter on Thursday November 22, 2012 @12:52AM (#4817665)
Has anyone ever seen DuckDuckBot hit their site? I don't have any web property large to appear via DuckDuckBot, but maybe someone else does? Im fairly certain it crawls Quora, as via this tweet https://twitter.com/yegg/status/33693491838066688
Re:Google competitor DuckDuckGo says it's getting shut out by ck2 on Thursday November 22, 2012 @12:58AM (#4817684)
It's not just you. Maybe he only crawls the Alexa top 10k or some similar minimal set.
Re:Google competitor DuckDuckGo says it's getting shut out by metalruler on Thursday November 22, 2012 @07:07AM (#4818464)
I checked my logs and there are several fetches from 72.94.249.37 and 72.94.249.38, over a number of domains that I host. None are particularly popular as far as the greater internet is concerned; one is a semi private site that I set up for my daughter's photos, another is one that has not yet been developed, apart from a few words of text and an image.

Interestingly, the fetches do not have a user-agent that identifies itself as the DDG crawler:

Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; .NET CLR 1.1.4322)

I'm assuming this is the crawler because it does not fetch anything besides text/html.

Re:Google competitor DuckDuckGo says it's getting shut out by Matt_Cutts on Thursday November 22, 2012 @11:37AM (#4819267)
That's interesting.

Gabriel, does DuckDuckGo's crawler have a distinct user agent? Can you talk more about how DuckDuckGo observes/respects robots.txt?

Re:Google competitor DuckDuckGo says it's getting shut out by boyter on Thursday November 29, 2012 @10:28PM (#4851680)
Follow up to this (its been a week so probably nobody will see this)

http://techzinglive.com/page/1028/179-tz-interview-gabriel-w...

around the 70 minute mark Gabriel mentions that DuckDuckBot is mostly about determining if pages are spam.

Re:Google competitor DuckDuckGo says it's getting shut out by samuellb on Thursday November 22, 2012 @12:28PM (#4819415)
No, it crawls non-Alexa sites too. I host two sites, one PR4 with Alexa rank 1.6+ million and another with PR3 and no Alexa rank. Only the latter one was accessed in the last year according to my servers access logs (and only with a single "GET /" request).
Re:Google competitor DuckDuckGo says it's getting shut out by elssar on Thursday November 22, 2012 @01:02AM (#4817696)
Try searching for something using their api, and compare with websearch. There is a world of difference between the search results. IIRC they aren't allowed to share results from some of their other sources via their api, so you can get a good idea of how much they take from other sources and how much from their own index.

But that is a non-issue as far as I'm concerned. As long as the results are relevant and they got them legally, then who cares where or how they came from?