I don't think the fact that the web site owner hasn't implemented better search than Google for their site is evidence that they can't do it, simply that they haven't.

This entirely. Big companies grow complacent. Which to me, in time, equates to Google's timely downfall.

No empire lasts forever; when was the last time you went to Sears?

Google knows about external inbound links, so has more data on which to rank your pages than you do.

I think the most important data that is difficult to obtain is a history of which links people click on for each search query term. I think that's really the special sauce that Google has that others do not. There is a catch-22 where you need a lot of users to get the data but you can't get the users without the data. Initially Google pulled ahead with inbound link counting, but anyone can clone that. It's the click data that's missing.

hah, that one is easy. People click the one at the top the most, then nr 2, after that nr 3, etc you see the pattern?

Google was epic when it started but quickly ended up indexing it self in a near closed loop. Everyone else was not important, it was all about the picture in front of them that was really just a mirror. Everyone stopped making home pages, forums and weblogs because it just doesn't make sense to do it.

Shit, people looking to buy something talking among their own before doing the purchase? KILL IT WITH FIRE!

Assuming someone has ever used that link you should have that data as well and with potentially even more relevance than google because you can see which pages are most often visited through external links.

my take is that the big bottleneck in search is not the index or the search but in understanding what it is the user is searching for in the first place.

that becomes a big can of worms very quickly, and starts to put into perspective some of googles other most successful verticals: NLP, Translation, Maps ("Near me"). Of course a lot of the verticals are also a generic means to get value out of all the server resources and experts required to operate the engine and then additional related branches to those verticals (email -> drive -> docs)

I guess the motivation is low as I frequently have to do this it seems.