But somehow, Google, as their last act of "don't be evil", decided to appease SEO people (who were a fiercely vocal crowd shitstorming over every change designed to counter their tricks) rather than ruthlessly pull through with the fight for result quality. If memory serves me right they were actively promoting "acceptable SEO" to lure practitioners away from even more annoying tactics. And then they failed to adapt once three "acceptable" started to ruin result quality. It's anyone's guess (perhaps even to Googlers who were right in the middle of is) wether that happened because of the slow-boil effect, commitment to their appeasing promises or because of plain old corporate complacency.

PS: oh, and in a certain way skewed search results even help Google, it increases ranking mysticism. When one site is consistently high, their competitors cannot know wether that is because of artful SEO or because heavy ad-spending is invisibly tipping the scales. So they will fearfully carry their own advertising budget to the altar of search ranking instead of running ads on a competing network like Facebook. Who knows wether The Algorithm considers ad-spend or not, better not provoke an angry god!

There's no such thing as "the whole web" - it is simply sites with the most money gaming Google better, and Google being happy for that to happen.

There's a lot else on the web, it just isn't surfaced by Google as much as in the search engine's early years.

That doesn't explain why DDG's results are improving, though.

This fails to consider the conflict of interest. The more 'commercialized' a site is, the more ad revenue they tend to bring in for google, so they have a perverse incentive to bring them traffic.

And yet people are happier with the results from competing search engines...