However, a user who just upgraded Windows and half their programs stop working will simply not upgrade. In many cases such dodgy applications are also never fixed because the vendor doesn't exist anymore, doesn't care, or the flaw is in some library they use and they don't really want to invest that much time to change it.
Besides, detecting such behaviour is pretty much the same as fixing it with a compat shim, since you have to do more or less the same work. So from MS's perspective the benefit of working around the buggy behaviour is much more prominent than trying to discipline developers by making users suffer.
There isn't a great answer to this that satisfies everyone.
If we don't push back against bad practices by developers we never improve the quality of the ecosystem.
In this specific instance, the user has a choice - don't upgrade.
That's how you get everyone stuck on Windows XP. There's no good choice for the user here–either you're on an old OS that's missing features or security updates, or you can't run software that used to work.
I regularly read The Old New Thing and just shake my head in wonder at how much more MS could have accomplished without that baggage.
I would much rather users were occasionally forced to upgrade their buggy software.
Yes, that's the ideal solution, but backwards compatibility overall is a huge can of worms that doesn't really have a good solution. In this case, it might not be possible to upgrade the software (e.g. it's unmaintained, or was contracted out, etc.)
The OS architecture no longer suffers from such things.
It’s better now - because they moved to abstracting a lot of that stuff as you said - but the number of man hours spent on it and the knock on effects of the overall system design is non-trivial.
Plus, if they’d not had that policy they could have been where they are now in the early 2000s
Kind of like OSX was.
Ignoring even the security implications, you can't simply not upgrade either as software doesn't exist in a vacuum. All the other software/hardware involved might need newer versions or bug fixes. Microsoft doesn't even support newer Intel chipsets in versions of Windows older than 10 -- that's only even possible because 10 is highly compatible with older versions of their OS.