I have already built similar large scale crawling systems using forks of Apache Nutch, Lucene and projects from hadoop eco system. To be honest with my limited knowledge, I just couldn't believe the numbers from the blog post. I came over to HN for smarter people to shed some light. But the master himself has revealed most of the answers. Thank you. We are not into what Ontolo does, that is just too complex for us.