Every distribution spends more time on packaging and integration testing than developing useful functionality. When developing for Linux you need to package your application in a few different formats. We also have different init systems to make it more difficult. Every distro has own store and repositories. For GUI there are different frameworks and display servers.
If Linux world standardizes around snap/flatpak, systemd and wayland then we stand a chance.
DE being the first thing a common user interfaces with, it holds a very important position, in my opinion. KDE and Gnome are very different w.r.t design principles, hence the polarization and debate. KDE and Xfce are not so different. I like Gnome, but have no use for it.
I agree with all else.
Also, I think, a real "free market" would resemble Linux ecosystem. No monopoly, small players and lots of choices. However, if we wish to compete in "real world", we need to do something similar to what you say.
Consider the possibility that your idea of what "the common user" wants is something you made up.
I would argue that having a choice between hundreds of DE's is a bad thing for the more casual user.
The large majority of their users don't even know what kernel is running the runtimes of their apps.
One of the reasons of netbooks failure, besides the cheap Windows XP licenses move from MS, was that each netbook Linux was its own little silo.
The big players tend to be compared, isn't that reasonable?
In spite of piracy, a large set of population actually pays for them, versus bashing the authors for considering to charge money for their work while suggesting free alternatives.
A full stack framework that in spite of OEM customization is relatively standard across all devices with an Android label on them.
I have helped people migrate from Windows to Plasma 5. They loved it. However, getting\installing software and general system management is still not straightforward enough. We also need system management tools that are solid. No terminal gymnastics at all for the common user on distros targeted at the masses.
Flatpak is all cool, but we need core user-space applications to be standardized. LibreOffice, for example, needs to become the industry standard office-suite. How can we achieve that? People I know just do not want to move away from MS Office. Even government agencies invest heavily in MS Office. This might just be the most important impediment.
I think, the Linux ecosystem shows us what a real "free market" looks like. Lots of small players and no monopoly. People love choices and they have. You want a minimal Window Manager, you got many to choose from. You want a full-blown Desktop Environment, you are spoilt for choice. Not a bad thing in an ideal world, but...