News for hackers. Stuff that matters.
Re:102TB of New Crawl Data Available by LisaG on Wednesday November 27, 2013 @06:08PM (#6811843)
There will be news about a subset sometime next month!
Re:102TB of New Crawl Data Available by malandrew on Wednesday November 27, 2013 @08:37PM (#6812524)
Ideally beyond the top sites, these subsets would be available as verticals, so that people can focus on specialized search engines.

While it's nice to have generalist search engines, it would be even better to be able to unbundle the generalist search engines completely. Verticals such as the following would be nice:

1) Everything linux, unix and both

2) Everything open-source

3) Only news & current events

4) Popular culture globally and by country

5) Politics globally and by country

6) Everything software engineering

7) Everything hardware engineering

8) Everything maker community

9) Everything financial markets

10) Everything medicine / health (sans obvious quackery)

11) etc.

Maybe make a tool that allows the community to create the subset creation recipes that perform the parsing out of data of a certain type and that the community forks and improves over time.

The time to create a generalist search engine has sailed, but specialist search engines is total greenfield.

Re:102TB of New Crawl Data Available by nl on Wednesday November 27, 2013 @10:10PM (#6812802)
You don't usually download this data - you process it on AWS to your requirements.

Seriously - they give you an easy way to create these subsets yourself[1]. That is a much better solution than them trying to anticipate the exact needs of every potential client.

[1] http://commoncrawl.org/get-started/

Re:102TB of New Crawl Data Available by malandrew on Wednesday November 27, 2013 @11:59PM (#6813095)
I guess what I was suggesting is "given enough eyeballs, all spam and poor quality content is shallow"

There is definitely a benefit in using the community to identify valuable subsets and then individually putting your energy towards building discovery/search products around that subset.

Re:102TB of New Crawl Data Available by gsnedders on Thursday November 28, 2013 @07:53AM (#6814308)
Is the example code still right with the new file formats for this new crawl?
Re:102TB of New Crawl Data Available by hkmurakami on Wednesday November 27, 2013 @06:39PM (#6812015)
would love to have even smaller subsets (like 5gb) that students can casually play around with too to practice and learn tools and algos :) (if it's not too much trouble!)
Re:102TB of New Crawl Data Available by Aloisius on Wednesday November 27, 2013 @08:04PM (#6812426)
You can fetch a single WARC file directly like say:

s3://aws-publicdatasets/common-crawl/crawl-data/CC-MAIN-2013-20/segments/1368704392896/warc/CC-MAIN-20130516113952-00058-ip-10-60-113-184.ec2.internal.warc.gz

They are around 850 MB each.

The text extracts and metadata files are generated off individual WARC files, so it is pretty easy to get the corresponding sets of files. For the above it would be:

s3://aws-publicdatasets/common-crawl/crawl-data/CC-MAIN-2013-20/segments/1368704392896/wat/CC-MAIN-20130516113952-00058-ip-10-60-113-184.ec2.internal.warc.wat.gz

s3://aws-publicdatasets/common-crawl/crawl-data/CC-MAIN-2013-20/segments/1368704392896/wet/CC-MAIN-20130516113952-00058-ip-10-60-113-184.ec2.internal.warc.wet.gz

Re:102TB of New Crawl Data Available by ccleve on Wednesday November 27, 2013 @09:42PM (#6812725)
Is there any way to get incrementals? It would be extremely valuable is to get the pages that were added/changed/deleted each day. Some kind of a daily feed of a more limited size.
Re:102TB of New Crawl Data Available by froo on Wednesday November 27, 2013 @11:27PM (#6813016)

  s3cmd ls s3://aws-publicdatasets/common-crawl/crawl-data/CC-MAIN-2013-20/segments/
That should get you about 90% on your way.
Re:102TB of New Crawl Data Available by alok-g on Wednesday November 27, 2013 @07:33PM (#6812293)
Totally true. Smaller versions are not helpful for just casual/student use; it also helps in code development and debugging. Otherwise, algorithm development gets impeded by scaling issues.
Re:102TB of New Crawl Data Available by ma2rten on Wednesday November 27, 2013 @08:07PM (#6812435)
Also interesting for machine learning. Where you can use it as a background collection.
Re:102TB of New Crawl Data Available by daivd on Thursday November 28, 2013 @10:41AM (#6814948)
One subset for each TLD would be nice. Or, if you can afford more CPU-power, per language, using a good open language detector.
Re:102TB of New Crawl Data Available by boyter on Wednesday November 27, 2013 @06:11PM (#6811861)
Fantastic news. Will be looking forward to seeing it.