News for hackers. Stuff that matters.
Re:CommonCrawl: an open repository of web crawl data that is... by abhishektwr on Monday December 12, 2011 @10:39PM (#3346173)
Just a pointer, the code for CommonCrawl Project is available on Github https://github.com/commoncrawl/commoncrawl
Re:CommonCrawl: an open repository of web crawl data that is... by pooyak on Tuesday December 13, 2011 @12:03AM (#3346377)
thread on HN from when common crawl was announced, interesting info there: http://news.ycombinator.com/item?id=3209690
Re:CommonCrawl: an open repository of web crawl data that is... by fungi on Monday December 12, 2011 @11:33PM (#3346304)
If you into said things then maybe http://yacy.net/ (p2p crawler and search) will be useful to you as well.
Re:CommonCrawl: an open repository of web crawl data that is... by Titanous on Tuesday December 13, 2011 @12:43AM (#3346476)
The latest data available is from 2010-09-25, which seems to be too old to be useful for most things.
Re:CommonCrawl: an open repository of web crawl data that is... by rgrieselhuber on Tuesday December 13, 2011 @12:27AM (#3346434)
It would be great to hear more about the tools they are using to crawl and potentially open it up to more people who want to contribute computing resources.
Re:CommonCrawl: an open repository of web crawl data that is... by emilis_info on Tuesday December 13, 2011 @06:51AM (#3347225)
This one may be also interesting for open data devs: http://scraperwiki.com/
Re:CommonCrawl: an open repository of web crawl data that is... by Aloisius on Monday December 12, 2011 @11:26PM (#3346285)
I hear a lot of people are crunching on CommonCrawl's data. It'll be interesting the type of stuff people come up with!
Re:CommonCrawl: an open repository of web crawl data that is... by nithinag on Monday December 12, 2011 @11:21PM (#3346269)
This looks really nice!