Re:How to write a crawler
by at-fates-hands on Sunday September 29, 2013 @06:59PM
(#6466894)
>>>You should always wait for several seconds (better yet, a minute) between requests to the same host.
This a thousand times.
I learned this lesson AFTER getting several angry emails from Admins and getting outright banned from one site for not having any delay between the requests in the first crawler I built.
This a thousand times.
I learned this lesson AFTER getting several angry emails from Admins and getting outright banned from one site for not having any delay between the requests in the first crawler I built.