News for hackers. Stuff that matters.
Re:102TB of New Crawl Data Available by nl on Wednesday November 27, 2013 @10:10PM (#6812802)
You don't usually download this data - you process it on AWS to your requirements.

Seriously - they give you an easy way to create these subsets yourself[1]. That is a much better solution than them trying to anticipate the exact needs of every potential client.

[1] http://commoncrawl.org/get-started/

Re:102TB of New Crawl Data Available by malandrew on Wednesday November 27, 2013 @11:59PM (#6813095)
I guess what I was suggesting is "given enough eyeballs, all spam and poor quality content is shallow"

There is definitely a benefit in using the community to identify valuable subsets and then individually putting your energy towards building discovery/search products around that subset.

Re:102TB of New Crawl Data Available by gsnedders on Thursday November 28, 2013 @07:53AM (#6814308)
Is the example code still right with the new file formats for this new crawl?