Re:How to crawl a quarter billion webpages in 40 hours
by TazeTSchnitzel on Saturday August 11, 2012 @12:14PM
(#4370366)
Also, some sites have a LOT of generated pages for a tiny section. For instance, crawling my GitHub account with my homemade crawler was impossible. There is an amazing level of depth there.
[ Parent
| on HN ]
There are no comments on this comment yet.