News for hackers. Stuff that matters.
Re:IBM Watson Acquires Technology from Blekko by boyter on Sunday March 29, 2015 @11:23PM (#9287513)
Perhaps. The problem is that crawling the web isn't that hard. Foreach(links) { get(link)} is all you really need. Doing it multiple times is harder though.

The real problem imho is building a distributed index and fighting spam. Both are incredibly hard problems to do well and very expensive. Hence so few are trying to do it.

Re:IBM Watson Acquires Technology from Blekko by metronius on Monday March 30, 2015 @04:30AM (#9288271)
If you would crawled web using Foreach(links) { get(link)} you would end up crawling useless automatically generated webpages.
Re:IBM Watson Acquires Technology from Blekko by psykovsky on Monday March 30, 2015 @07:18AM (#9288768)
You say that like it doesn't happen all the time. googlebot included.
Re:IBM Watson Acquires Technology from Blekko by bobajeff on Monday March 30, 2015 @12:21AM (#9287637)
I think providing quality results is the challenge. That take certain algorithms and optimizations to do so in a timely fashion. This is where AI projects like Watson really come into play add they are all about retrieving the correct information (a difficult problem on its own) in a timely fashion.
Re:IBM Watson Acquires Technology from Blekko by boyter on Monday March 30, 2015 @01:03AM (#9287740)
Also true, however I think a basic ranking algorithm should still be fine for most cases if you have 100% spam eradication. Good content left over should bubble to the top.

I tried building a small search engine with a very basic algorithm and it worked very well for 90% of searches.

I agree with you in part but I also believe if you got the others 100% right you would have a respectable engine by itself.