But is scalability a computational complexity problem for you? Or you think you can search the whole web like that?

Moore's law and general improvements to the software mean that even if our algorithm is as expensive as people think it is, we'd be fine.

That said, scaling it isn't trivial and I don't want to sound overly dismissive. :)

I firmly believe that we can search the whole web this way, but Powerset doesn't have the hardware assets at the time I write this post to do the whole web (or we would have done it).

I'm not sure you can pull the Moore's Law card in this case, since I imagine the size of the web has grown at least as fast as processors have improved.

I can't say I know how fast the web is growing. I do have some idea of how fast Yahoo! and Google's search indicies are growing, and they aren't that exponential curve.

It's never going to be the case that the PARC algorithm we use will be as fast as Google's algorithm. There is simply a lot more work to do, so they stay a fixed width apart.

But the way that the industry looks at computing resources has changed a lot since when Google is founded. They're becoming a commodity to the point where you can rent CPUs on a hourly basis. Also, there are new kinds of hardware that are starting to be tapped (for example, over the last few years graphics cards have become more and more a platform for general parallelized computation) that make the task seem feasible.