jmischel
Re:The Spider of Doom
on Wednesday October 12, 2011 @04:01PM (#3104310)
I ran into this with my crawler, too. See http://blog.mischel.com/2008/08/08/hey-you-deleted-my-files/
Web-scale crawling at cable modem speeds
http://blog.metadatalabs.com/why-focused-crawling
A high-level overview of why and how we use machine learning techniques to focus our crawling, giving us wide coverage of online video with limited bandwidth.