News for hackers. Stuff that matters.
Re:IBM Watson Acquires Technology from Blekko by mark_l_watson on Sunday March 29, 2015 @10:23PM (#9287332)
A good reason to use and support Common Crawl - a good resource, that is updated (I think) every few months).
Re:IBM Watson Acquires Technology from Blekko by thomasfoster96 on Sunday March 29, 2015 @11:13PM (#9287489)
Cuil (ha!) donated most of their crawled data to the Internet Archive - so with that plus Common Crawl, it's pretty easy for anyone to have a sizeable index.

What those neither of those give you is an up to date index, which is why small search engines still need Yahoo and Yandex's APIs. I'm not sure any free resource can match the speed at which big companies can index the web.

Re:IBM Watson Acquires Technology from Blekko by sytelus on Monday March 30, 2015 @02:52AM (#9288002)
Common Crawl is more interesting for research purposes than for real search engine business. It refreshes only 4 times a year and has only 2 billion pages. For anything under ~10B pages is most likely to be called "research" or "toy" index. You have almost zero chance of building competitive index if you are below ~15B. Google is rumored to crawl about 150B pages as reference. There are estimated 1.2T pages out there which does not get crawled due to various reasons.
Re:IBM Watson Acquires Technology from Blekko by metronius on Monday March 30, 2015 @04:45AM (#9288331)
I dont think that you need so much pages - most of 150B pages will be never ever shown displayed at SERP.
Re:IBM Watson Acquires Technology from Blekko by Arnt on Monday March 30, 2015 @10:32AM (#9289737)
True, you just need a subset. Now how you you identify that subset without indexing the pages to find out whether each page is in the subset you need?

IIRC google used to scan different pages at very different frequencies. Quite possibly because it has assigns pages into subsets every time it indexes.

Re:IBM Watson Acquires Technology from Blekko by boyter on Sunday March 29, 2015 @11:23PM (#9287513)
Perhaps. The problem is that crawling the web isn't that hard. Foreach(links) { get(link)} is all you really need. Doing it multiple times is harder though.

The real problem imho is building a distributed index and fighting spam. Both are incredibly hard problems to do well and very expensive. Hence so few are trying to do it.

Re:IBM Watson Acquires Technology from Blekko by metronius on Monday March 30, 2015 @04:30AM (#9288271)
If you would crawled web using Foreach(links) { get(link)} you would end up crawling useless automatically generated webpages.
Re:IBM Watson Acquires Technology from Blekko by psykovsky on Monday March 30, 2015 @07:18AM (#9288768)
You say that like it doesn't happen all the time. googlebot included.
Re:IBM Watson Acquires Technology from Blekko by bobajeff on Monday March 30, 2015 @12:21AM (#9287637)
I think providing quality results is the challenge. That take certain algorithms and optimizations to do so in a timely fashion. This is where AI projects like Watson really come into play add they are all about retrieving the correct information (a difficult problem on its own) in a timely fashion.
Re:IBM Watson Acquires Technology from Blekko by boyter on Monday March 30, 2015 @01:03AM (#9287740)
Also true, however I think a basic ranking algorithm should still be fine for most cases if you have 100% spam eradication. Good content left over should bubble to the top.

I tried building a small search engine with a very basic algorithm and it worked very well for 90% of searches.

I agree with you in part but I also believe if you got the others 100% right you would have a respectable engine by itself.