News for hackers. Stuff that matters.
Re:The Perils of Web Crawling by redstripe on Friday December 10, 2010 @05:32PM (#1993454)
"Sites will ban you"

Wish it would happen here on HN. Seems like it's a weekly occurrence where someone announces a pet project that involves crawling the whole site. I can't help but to think this is connected to the 30+ second page loads I get here often.

There are no comments on this comment yet.