Re:Ask HN: How Much Can I Scrape?
by lacker on Tuesday November 11, 2008 @02:43AM
(#360344)
If you contact someone at Yahoo, the response will be, do not scrape us in any way.
The problem with financial data is that Yahoo (like most other sites where you might find this data) doesn't generate this data themselves. They license it from other companies, and the licensing agreement typically prohibits or greatly restricts Yahoo's ability to provide the data to third parties.
That said, if Yahoo is not aware that you are scraping them, they cannot stop you. They certainly do have anti-scraper algorithms (you will start getting http 999 errors) but they will not kick in until you cross some invisible threshold. You can probably use Tor with no problem.
Although, if you get large enough that someone notices, you will probably get some sort of cease & desist letter. Depending on your goals that might not be a problem for you.
The problem with financial data is that Yahoo (like most other sites where you might find this data) doesn't generate this data themselves. They license it from other companies, and the licensing agreement typically prohibits or greatly restricts Yahoo's ability to provide the data to third parties.
That said, if Yahoo is not aware that you are scraping them, they cannot stop you. They certainly do have anti-scraper algorithms (you will start getting http 999 errors) but they will not kick in until you cross some invisible threshold. You can probably use Tor with no problem.
Although, if you get large enough that someone notices, you will probably get some sort of cease & desist letter. Depending on your goals that might not be a problem for you.