News for hackers. Stuff that matters.
Re:Interesting Data Sets for Statistics by sytelus on Friday May 30, 2014 @12:39AM (#7820298)
Surprisingly it doesn't mention HN itself which is a treasure trove of data. I know there is APIs to download HN content but is there a permanent location for HN data dump (like StackOverflow do their data dump on Internet Archive)? This is a great article, BTW, anyone who wants some cool projects to do in data mining and machine learning.
Re:Interesting Data Sets for Statistics by j2kun on Friday May 30, 2014 @10:01AM (#7821815)
I currently have a subset of one year's worth of all HN stories and comment trees, organized by story, but it's on my local machine. Where is a good place to post it? It's quite big, on the order of multiple GB.

The problem (if you want an easy scraper) is that the HN API limits you to 1k requests per hour. So it took me about 10 days of continuous running and restarting because of random crashes to get all the data.