News for hackers. Stuff that matters.
Re:Scraping, cleaning, and selling big data by rauljara on Wednesday May 11, 2011 @01:51PM (#2537436)
I recently had to scrape the yellow pages for a list of doctors around chicago. My first script caused them to block my ip address. The warning page was cute, it said I had violated one or more of Asimov's 3 laws of robotics.

I wrote a second script that included delays between loading pages (just 10-15 seconds, some longer pauses every 10 pages or so) and it ran for a whole day without being blocked.

There are no comments on this comment yet.