Don't crawl the site, please. The place to get data is the HNSearch API.
Re:The island where people forget to die
by ippisl on Wednesday October 24, 2012 @04:58PM
(#4695019)
There's a database(quite old) of HN posts and comments here:
http://www.btscene.eu/details/2240774/Hacker+News+Database+o...
Re:The island where people forget to die
by malandrew on Wednesday October 24, 2012 @05:06PM
(#4695060)
Any plans to try to get a dataset for supervised ml? Perhaps collect the top 4 comments from all front page threads and post a survey on HN asking HNers to rate those comments for skepticism/dismissiveness?
To any that have experience getting comments data from HN -- what's the fastest, most polite way to do this? And am I correct in remembering that there's some aggressive rate-limiting for crawling the site?