I think 60% of my searches are now done with "site:" on google and the results are way better than most of the onsite searches.
For your second point what I know is that there has been 4 decades of research into search relevance ranking and about 0% of that knowledge has made it into real life products.
I went through a phase where I interviewed people at 15 or so search vendors (e.g. Open Text) and not one of them told me that they or their customers thought relevance was a priority. One of them participated in the TREC competition because they thought it would give them some credibility but then they decided it didn't.
If you look at any of those products you find they have hundreds of low-quality "integrations" with various data sources because they believe that's what sells the product.
Google keeps competition away because it is able to generate much more revenue per search than competitors. Worse results translate to worse revenue (e.g. if the #1 result was always the best, why look at the ads?)
If somebody else could make the money work they could definitely give better search results.