News for hackers. Stuff that matters.

mixnode

Re:A web crawler that can handle any number of websites on Wednesday April 12, 2017 @04:13PM (#14101305)
We've done crawls of over 250TB and 3 billion web pages. If you need to crawl more than this you can simply shoot us an email and we'll provision resources for your mega crawl.
Very interesting. I wonder why it's not getting more upvotes :)