Re:The Perils of Web Crawling
by redstripe on Friday December 10, 2010 @05:32PM
(#1993454)
"Sites will ban you"
Wish it would happen here on HN. Seems like it's a weekly occurrence where someone announces a pet project that involves crawling the whole site. I can't help but to think this is connected to the 30+ second page loads I get here often.
Wish it would happen here on HN. Seems like it's a weekly occurrence where someone announces a pet project that involves crawling the whole site. I can't help but to think this is connected to the 30+ second page loads I get here often.