Every story and comment from the Hacker News API, in a layout borrowed from 2001.
A comment on Scraping, cleaning, and selling big data. Parent comment
http://www.gabrielweinberg.com/blog/2010/08/thoughts-on-yaho...
-----------------
and check http://www.reddit.com/r/datasets and infochimps, maybe dataset is already out there
--------------------
and ...never spider from home, you'll always screw something up: referrer HTTP header, user agent, random wait time between requests,
http://streamhacker.com/2010/10/04/perils-web-crawling/
http://news.ycombinator.com/item?id=2463058 see first comment
There are no comments on this comment yet.
http://www.gabrielweinberg.com/blog/2010/08/thoughts-on-yaho...
-----------------
and check http://www.reddit.com/r/datasets and infochimps, maybe dataset is already out there
--------------------
and ...never spider from home, you'll always screw something up: referrer HTTP header, user agent, random wait time between requests,
http://streamhacker.com/2010/10/04/perils-web-crawling/
http://news.ycombinator.com/item?id=2463058 see first comment