Re:IBM Watson Acquires Technology from Blekko
by metronius on Monday March 30, 2015 @04:30AM
(#9288271)
If you would crawled web using Foreach(links) { get(link)} you would end up crawling useless automatically generated webpages.
Re:IBM Watson Acquires Technology from Blekko
by psykovsky on Monday March 30, 2015 @07:18AM
(#9288768)
You say that like it doesn't happen all the time. googlebot included.
Re:IBM Watson Acquires Technology from Blekko
by bobajeff on Monday March 30, 2015 @12:21AM
(#9287637)
I think providing quality results is the challenge. That take certain algorithms and optimizations to do so in a timely fashion. This is where AI projects like Watson really come into play add they are all about retrieving the correct information (a difficult problem on its own) in a timely fashion.
Re:IBM Watson Acquires Technology from Blekko
by boyter on Monday March 30, 2015 @01:03AM
(#9287740)
Also true, however I think a basic ranking algorithm should still be fine for most cases if you have 100% spam eradication. Good content left over should bubble to the top.
I tried building a small search engine with a very basic algorithm and it worked very well for 90% of searches.
I agree with you in part but I also believe if you got the others 100% right you would have a respectable engine by itself.
The real problem imho is building a distributed index and fighting spam. Both are incredibly hard problems to do well and very expensive. Hence so few are trying to do it.