News for hackers. Stuff that matters.
Re:A Look Inside Our 210TB 2012 Web Corpus by greglindahl on Tuesday August 13, 2013 @11:25PM (#6209784)
blekko attempted to implement crawling a local copy, and it was a PITA. We'd rather crawl the real thing with a crawl-delay of 1. Best would be if the Wikimedia Foundation made a .html dump available.

There are no comments on this comment yet.