News for hackers. Stuff that matters.
Re:How to crawl a quarter billion webpages in 40 hours by TazeTSchnitzel on Saturday August 11, 2012 @12:14PM (#4370366)
Also, some sites have a LOT of generated pages for a tiny section. For instance, crawling my GitHub account with my homemade crawler was impossible. There is an amazing level of depth there.

There are no comments on this comment yet.