News for hackers. Stuff that matters.
Re:Ask HN: How Much Can I Scrape? by oakmac on Tuesday November 11, 2008 @02:33AM (#360334)
I did this exact same thing a few years ago only stripping the data from nasdaq.com using Perl. I used to hit their site once every 2 seconds times roughly 2500 stocks every day for about 6 months. I would only grab as much data as I needed. They never contacted me or blocked my IP address. I also had a friend who was doing the same thing for a longer period of time.

From experience, I would not recommend getting your data from Yahoo. I looked at them first, but their data is just not as good as the source.

If you would like more information or my notes on how I reverse-engineered the nasdaq.com URL scheme please send me an email.

Re:Ask HN: How Much Can I Scrape? by oakmac on Tuesday November 11, 2008 @11:33AM (#360616)
I updated my profile with my email address.