crawling the web is only easy if all you ever do is following hrefs... we're in the world of single-page turing-complete web apps and you're underestimating the amount of pure compute, ingress and egress bandwidth necessary to keep the index up to date. there's a reason google has designed and built their data centers, their server chassis and even power supply.