News for hackers. Stuff that matters.
Re:How to write a crawler by at-fates-hands on Sunday September 29, 2013 @06:59PM (#6466894)
>>>You should always wait for several seconds (better yet, a minute) between requests to the same host.

This a thousand times.

I learned this lesson AFTER getting several angry emails from Admins and getting outright banned from one site for not having any delay between the requests in the first crawler I built.

There are no comments on this comment yet.