News for hackers. Stuff that matters.
Re:A Look Inside Our 210TB 2012 Web Corpus by mark_l_watson on Tuesday August 13, 2013 @07:20PM (#6208958)
Check out the Common Crawl contest winning projects from the linked page - some very good work, and a good source of ideas and techniques: http://commoncrawl.org/the-winners-of-the-norvig-web-data-sc...

Some good stuff!

Re:A Look Inside Our 210TB 2012 Web Corpus by wicknicks on Tuesday August 13, 2013 @09:31PM (#6209483)
I loved the inter-lingual web page linkage visualization project. Any idea why Traitor won the contest? It seems very similar to regular "create inverted index with map reduce" problem, or am I missing something?
Re:A Look Inside Our 210TB 2012 Web Corpus by mark_l_watson on Wednesday August 14, 2013 @02:02PM (#6213147)
Perhaps Traitor won because it is such a good example of using Map Reduce over the Common Crawl data? I agree that inter-lingual was a cool project.