News for hackers. Stuff that matters.
Re:Lessons Learned While Crawling the Web by alecsmart1 on Friday October 04, 2013 @02:18PM (#6496995)
The title is misleading. It should be "Lessons learned about web crawlers while crawling the web"
Re:Lessons Learned While Crawling the Web by jgmmo on Friday October 04, 2013 @04:25PM (#6497709)
I think it should be called 'web scraping 101'

Very thin article.

Re:Lessons Learned While Crawling the Web by billrobertson42 on Saturday October 05, 2013 @01:54PM (#6501419)
I found certain aspects of it really interesting. In particular, the way that the author interacts with the crawling software to gain insight into the structure of a website. Not just the process, but the way he interacts with it is interesting too.