Re:A web crawler that can handle any number of websites
by mixnode on Wednesday April 12, 2017 @04:13PM
(#14101305)
We've done crawls of over 250TB and 3 billion web pages. If you need to crawl more than this you can simply shoot us an email and we'll provision resources for your mega crawl.