It really seems high-quality search is fundamentally in opposition to serving ads, alas. (At least once every page in existence probably serves ads via the search operator's network.)

I have a lot of complaints about, say, McDonald's, but they make their money through giving people something of value to them. Advertising legitimizes the making of money in a way unrelated to value delivery. (The same is true about a lot of finance.)

When you combine that with up-and-to-the-right numerical goals and standard executive incentives, over time you pretty much guarantee what Doctorow calls "enshittification". Delivering value becomes at best a side effect of the system.

"[W]e expect that advertising funded search engines will be inherently biased towards the advertisers and away from the needs of the consumers"

and

"[W]e believe the issue of advertising causes enough mixed incentives that it is crucial to have a competitive search engine that is transparent and in the academic realm."

Both from The Anatomy of a Large-Scale Hypertextual Web Search Engine by Sergey Brin and Lawrence Page (1998)

... but more seriously: Yeah, their reasoning (now) isn't exactly going to be unbiased and/or unblemished by their own experience. They probably have the most extreme survivorship bias ever. (Not their fault, they just do because of whatever factors got them into the position they're in.)

There are, of course, a few relatively successful paid general purpose search engines but these serve a niche demographic if you consider the world-wide scale of google et al. Possibly specialized search (we build one) will be able to thrive in the future, but these engines also serve niche markets in the end.

Thus, the real competition to ad-based search is not high quality search and that is likely why search results don't get better.