If you're going to compete with Google as the one-stop-shop for searches, you're going to need to be able to provide search results that are consistently competitive. And I think Google has gotten good enough that you simply can't do that without access to some portion of their moutains of data. Nobody else has search history for every single American, and without that you are going to be crippled when {Random American} tries to search for something.
I observed there is a set of results that can be found in all engines but each engine has a smaller set that are not found on the others. For this variety of results I occasionally use both Google and Bing.
This variation algorithm that I do, is one that yields better results then sticking with the brute force of either one.
It is these "getting out of the box" algorithms that out compete brute force algorithms on the same ground. That means, with guided context your search results improve drastically.
Agreed. An easy way to test the difficulty of this is to switch to Bing for a week. I actually tried!
I've seen Norvig's video myself and while I see that they get better results with brute force on more data, that does not mean that better algorithms don't exist.
In fact, I know for sure there are better algorithms (only because we have for example humans that are better than AI on certain tasks) that probably can use the available data in a more intelligent way than brute force, yielding better results in the process.