The real problem lies in this practice. Which other popular system uses this approach? Windows, macOS but also open source alternative like FreeBSD all have well-defined kernel API and drivers SDK. So, not providing a standard interface but then crying for the lack of third party drivers is hypocritical: the problem is self-inflicted.
No other popular system follows the same development cycle as the Linux kernel.
There's clearly a non-zero (and indeed non-trivial) cost to providing a stable kernel driver API, and the kernel devs don't want to bear that cost, as they feel it would hold back the kernel from evolving, improving on APIs, and fixing security issues or other implementation bugs.
The other side of the coin is that closed driver developers want this stable API to reduce their own development costs, but they will not follow the "intended" workflow that reduces the ongoing maintenance cost to effectively zero.
"Pro" move is to just skip the costly maintenance and keep the kernel same, for old times sake.
Joking aside, I think a lot of this comes down to the incentives (or lack thereof) in the ecosystem - there is no real penalty for releasing an insecure product, or one that can't be kept up to date. Thus inventivising shipping on an outdated kernel you can't maintain, and never patching it.
If those shipping products had to factor in the externalities of this decision, they would rapidly be mainlining everything to reduce the costs of this. The cost of "mainlining" their code will be trivial compared to the (unmeasured) cost of the disposal of the equipment during a reasonable lifespan due to lacking updates, and the replacement of the equipment.
Every release, there's refactoring and improvements of the kernel subsystems. They don't offer stable APIs (by design) for drivers - you need to keep it up to date.
If you want to see this for real, try getting a "code thrown over the wall once" type embedded device kernel tree, figure out the kernel version it shipped on (likely ancient), and try to apply the latest patch level for that release to it. Then resolve the inevitable issues even just applying a few minor bug fixes.
Now if you to to update to the next major kernel version, you'll start to find functions your driver tried to call aren't there, or have new function definitions. You need to dig through the code to see what's changed. Over a few major versions, you'll eventually realise this is a whole lot of bother. If you just mainline your driver, you'll no longer need to worry about this costly maintenance burden - it might well be a few engineers spending most of their time on it. That quickly adds up, and suddenly mainlining starts to make sense.
Nvidia has to keep updating their proprietary driver every release, and presumably has a few developers doing this. I can't speak for how well it works, as I always go for AMD, but other comments here seem to confirm distributions will happily break Nvidia drivers because they aren't mainline and therefore aren't tested.