News for hackers. Stuff that matters.
Re:Ask HN: What's your philosophy on page scraping? by rmanocha on Friday March 26, 2010 @06:22AM (#1220675)
I think that if you're generally respectful of the target websites - scraping them is ok. For example, I scrape various government websites for my website. I use a random delay between requests and am generally very careful about not requesting the same page multiple times (this is hard 'cause a lot of the pagination happening on these pages is via JS calls).

I am ok if someone decides to scrape my websites in a similar fashion - although if I do see that starting to happen, I'd rather just go ahead and build an API.

There are no comments on this comment yet.