Re:Lessons Learned While Crawling the Web
by alecsmart1 on Friday October 04, 2013 @02:18PM
(#6496995)
The title is misleading. It should be "Lessons learned about web crawlers while crawling the web"
[ Parent
| on HN ]
I think it should be called 'web scraping 101'
Very thin article.
Re:Lessons Learned While Crawling the Web
by billrobertson42 on Saturday October 05, 2013 @01:54PM
(#6501419)
I found certain aspects of it really interesting. In particular, the way that the author interacts with the crawling software to gain insight into the structure of a website. Not just the process, but the way he interacts with it is interesting too.