AFAICT, quality personalized search results isn't the goal for Google here.

I think the point of all that is generating revenue via advertising sales.

And while providing high quality search results might once upon a time have been a goal, both as a goal in itself and a tool to drive user adoption/engagement, that's no longer necessary as they have a (relatively) captive audience and a (relatively) captive customer base (advertisers). As such, quality search results are no longer all that important.

I'm not a Googler, IMHO, YMMV, etc.

Now they have millions of cell phones collecting data and the GPS location of everyone offline, they run extremely popular DNS servers to see what websites people go to, most websites (including educational and government websites) include google's trackers so on most websites every page loaded will ping at least one of google servers. They've got people uploading their personal and work documents to their cloud. They are swimming in data collected from sources outside of web search. Between that and the lack of actual competition it's no surprise they aren't investing in making searching the internet better for people. In some ways they profit from search results being terrible. If anything beyond the first few results on the first page is filled with spam and irrelevant websites the top few results to any search become even more valuable.

That depends on whether or not you agree with Google's definition of "better". What is better for Google is a set of results that make you most likely to click on an advert. Failing that, they want you to click a link, see the site, and either click an ad there or quickly return to Google where you'll click on an ad. The worst outcome for Google is that you'll click a non-paid search result, like it, and stay on that website.

This is the fundamental problem with search engines - if they work well and give you exactly what you're searching for first time then they won't make any money. A lot of what Google does is subtly trying to give you results that look great but really aren't.