News for hackers. Stuff that matters.
Re:DuckDuckGo Architecture - 1 Million Deep Searches a Day a... by afsina on Monday January 28, 2013 @02:59PM (#5130315)
Sadly, I do not have that experience. I know what everybody knows. I guess it is extremely hard. Software-Infrastructure wise it is possibly one of the most challenging product. Competition is stiff, so going global is almost not a choice. You either go vertical, or local-national. Yes you can use available open source software (Nutch, Hadoop, Solr etc) but they can make you go up to a point. If you want to go real big, then you may need to write your own crawlers, indexing - ranking, clustering, duplicate-spam detection algorithms, distributed file systems, ad systems etc. Making a search engine of course is not enough, you need to add a gazillion service to make it attractive. Many of those services are also requires cutting edge technology (translation etc.). And it burns money 7/24.

I understand why people values DuckDuckGo for privacy reasons. I respect that. But sure devleoping it, is far far easier than developing a real search engine.

Re:DuckDuckGo Architecture - 1 Million Deep Searches a Day a... by greglindahl on Monday January 28, 2013 @03:07PM (#5130361)
If you're interested in what it takes to write a search engine with its own multi-billion-webpage crawl/index, check out the blog series starting with http://highscalability.com/blog/2012/4/25/the-anatomy-of-sea...
Re:DuckDuckGo Architecture - 1 Million Deep Searches a Day a... by gummydude on Monday January 28, 2013 @07:31PM (#5131644)
And when they said they're moving towards knowledge graph like instant results, latency not gonna be pretty, lots of small fetchings after general queries from core search servers.