I think we need to optimize local search engines and aggregate (in some ways) all this into a global search engine.

https://www.google.com/work/search/products/gss.html

I used it several years ago to power this site search:

http://www.poetryfoundation.org/search/articles#qs=ginsberg

It works pretty well (disclaimer: I'm not sure this is still being used on that site)

Since we had all the data the results from Google were compared to a local search I setup in Sphinx (http://sphinxsearch.com/) and the Google results were more relevant and it was a lot cheaper to deploy.

Full-text indexing (what ES provides) has been around for almost forever, ES just does a way better job of productizing/delivering it.

However, Google is far more than a text index. Ranking, currently is still very difficult and requires messing around with facet and weighting parameters.

In my opinion which means nothing, sites need to figure out how to power their own search. Using a third party isn't going to work for most. Maybe people need to focus on building custom architecture that indexes the data in a more structured way rather than cobbling systems together that ultimately hinder search efforts when its time to get the user what they want. I don't know the answer but somebody eventually will. Maybe wordpress will create a powerful search for all those wordpress sites.