News for hackers. Stuff that matters.
Re:Download the internet by snprbob86 on Tuesday February 03, 2009 @10:46PM (#465224)
The biggest problem here is hosting the index... in RAM...

It takes A LOT of machines to power a modern search engine which serves any real amount of traffic. One key component of an open source search engine would be a sort-of peer-to-peer distributed infrastructure. When I suggested this in an earlier thread, people were quick to point out the liability concerns here... but maybe it could work somehow... but then how do you get people to sign up for it?

That said, I think this is incredibly interesting stuff. I would really love to see open source, peer-served web utilities. For example, I'd want access to many of the components of a web search engine, not just the search results themselves. Things like a language model for spell checking or word segmentation. Or a set analysis tool for detecting synonyms.

Re:Download the internet by braindead_in on Tuesday February 03, 2009 @11:08PM (#465244)
Ya. Thats true. But purely from an data standpoint, having the same index as Google has would be valuable. Just because you can do so many things with it. Like, figuring out how to increase the page rank of your site!