So... why upgrade in the first place then, to solve what? Something I learned as a software developer is that if a piece of code is working properly, the best thing you can do is to leave it alone.
Exactly! And that's what both the corporate executives and your parents are thinking!
So Microsoft has always had to work hard to convince people to upgrade. New features were the natural way of doing that. But not maintaining backward compatibility, in a world where almost all software was an one-time purchase, would be a deal breaker. Why upgrade if it's going to cost $20M to make our ERP suite compatible with the new Windows? Why upgrade if all the software I spent $1000 on over the past few years will no longer work? Hence backwards compatibility with existing software was a top priority (to the point that Windows has a lot of fixes to badly coded third-party software built in).
Because they make a lot of money on enterprise customers. The only way to convince them to upgrade is to guarantee the new OS version won't break their legacy software. But even on consumer market, in the 90s and 2000s it was important old videogames work on new versions of Windows, or otherwise people would not upgrade.
This comes from before the days of web applications and everything as a Service. Software vendors had to work hard to get people to pay them again.