Before Google's page rank algorithm there was a lot of research into document search. A favorite of mine was 'scatter gather'[1].

I often wanted ability to filter and group when staring a pages of results from redhat jboss support forums when trying to fix a dead jboss cluster. But I quit that job so haven't had the need recently.

Edit; point being it would be nice if someone came up with a service that implemented the ideas in this paper :)

[1] http://www-users.cs.umn.edu/~han/dmclass/scatter.pdf

Is there any chance we might see an updated index for the Common Crawl? I've tried using the dataset before, but I found it difficult given that you have to process the entire thing in order to find the particular pages you are looking for.

As an example, I was trying a project to look at the top news sites, like BBC, CNN, Al Jazeera, etc. Then matching articles about the same news topic on each site, before finally fact checking for differences between the stories (ie. 20,000 homes were without power vs. 50,000)

That kind of project requires a load of crawling, but I can't look for specific pages without processing the entire CC set first.

I love the project though, so thank you for doing it!

So awesome to see that folks are discovering Common Crawl, it's a really great project and the amount of data they've crawled has been tremendous. If you're looking to get into machine learning, it's perhaps the most comprehensive dataset you can get your hands on.

anyway we can get an open sourced version of that optimized job for us to play with?

There's are two levels of optimizations that come into play: AWS setup and the choice of primary language. I'm always happy to speak about both as we love seeing experiments run over the data!

AWS optimizations: For that level of cost efficiency, you really need to use spot instances. A cluster of 100 m1.xlarge machines (1.5TB of RAM, 400 cores, and 168TB of magnetic disk storage) will only cost you $3 per hour using spot instances, rather than $30 on-demand. You should pay on-demand prices for the Hadoop master however.

You'll also want to roll your own Hadoop cluster as opposed to using Elastic MapReduce (EMR). EMR is amazing but the cost overhead when using it on spot instances is ~100%.

For the code itself, this is a situation where you'll want to stick with the programming language that has both the best performance and best ecosystem. I'm personally not a big fan of Java, but it really does win out here -- it's close to C or C++ for performance and has the advantage of the Hadoop ecosystem behind it. Other languages are certainly usable, but even if LanguageX only ran 4x slower than Java, the resulting job would be 4x more expensive due to paying by the hour.

Other than that, it's really just a standard MapReduce job using Hadoop. You can see an three examples for the three different data formats we use at: https://github.com/commoncrawl/cc-warc-examples/

Thanks for your comments. I see you get 400 cores for 3USD/h . Where you optimizing for processing power per dollar? Do you know if the m1.xlarge instances are the best for this use case?

It really depends on the use case. I mentioned m1.xlarge as they provide a good general cluster setup. The mix of CPU, RAM, and disk space should work well for most experiments one might want to perform. m1.xlarge instances also have 1Gbps network interfaces when others in the same price range have 500Mbps -- a vestige of the older generation of machines. Finally, they excel at disk space. If you're utilizing HDFS heavily, newer instances are usually SSD (good) but have 10 to 20 times less disk storage (bad).

For the same dollar amount, you can trade for other specs though.

If you're more interested in CPU / RAM / SSD for example, paying $3USD/h for r3.xlarge gets you 3TB of RAM, about 1.5 times more computing power (same number of cores but more compute units), but far less disk space -- 8TB of SSD.

In the end, it really depends on the task at hand, but your dollar does go quite far regardless!

what would be the benefit of having the web as a dataset when it is rife with copyright laws (see craigslist), monopoly businesses who viciously protect their human uploaded content and profiles, and majority of the data from the web being useless without a context and purpose of the searcher?

I'm just curious as to how commoncrawl compares with kimonolabs and import.io as they seem to have the same goal of creating an internet as a dataset, or an API. I can't help but feel like it's just solving another 'semantic web' problem that nobody asked for.

It is funny that the most demanding customers of semantic web are also the ones who are willing to pay the least amount of time and money.

Regarding copyright issues, I believe you can still use copyrighted data as long as it's transformed. E.g., building language models, or doing a search engine like Google. In fact, I can think of more computational uses for copyrighted data, while on the "banned" side, I can only think of... SEO.

Regarding point two: "monopoly businesses who viciously protect their human uploaded content". I spend a lot of time scraping these monopoly businesses, and it seems to me they do a decent job of letting their users decide what data is exposed. Facebook, linkedin, and google all are decent about letting me scrape their public info. That's all I have a right to -- private info should stay private, at the behest of the owner (the User in UCG).

You are correct regarding the third point, but I don't see that as a problem. This isn't a solution in search of a problem -- it's a problem without a solution at the moment.

Here's a toy example of something I'd like to do: calculate the positive / negative sentiment of commenters at particular baseball fan sites, so I can hide the content I don't like, and show that which I do. Having a common crawl of the site would be immensely useful (and is indeed a prereq) for this. I wouldn't need to republish it, just compute on it.

+1