Re:Lessons Learned While Crawling the Web
by billrobertson42 on Saturday October 05, 2013 @01:54PM
(#6501419)
I found certain aspects of it really interesting. In particular, the way that the author interacts with the crawling software to gain insight into the structure of a website. Not just the process, but the way he interacts with it is interesting too.
Very thin article.