We've seen this with Python2->3 transition. If the new version isn't backwards compatible (requires non-trivial work to transition), what you've done isn't an upgrade: you've created a new product, and deprecated the old one. As we've seen from Google, people hate it when that happens (how many different eras of deprecated chat app are Google on now?)
Other examples: Windows has, on a couple of occasions, attempted architecture transitions. Windows RT was the latest. They've not yet made the leap to ARM, despite having cutdown versions of Windows (CE) running on ARM for something like 20 years. Apple have managed multiple architecture transitions by providing emulators, bridges, "fat binaries" that work across the transition, and a history of being willing to tell both developers and customers to get stuffed if they don't like it.
Not only that, but breaking backwards compatibility will cause dependency conflicts once a dependency graph is a bit larger, and this can easily completely stall transitioning of software to a new version.
We are talking about an API, not a shared service.
That distinction is becoming a lot blurrier than it used to be as more and more "APIs" now are run as shared services with no option of running them locally.
Backwards compatibility is better than any other option. From time to time you need to provide new, clean versions of old APIs and that's it. Things get deprecated (as in "best not to use this unless you are aware of the consequences". Aggregate over time, don't replace. Garbage collect when not reachable. This works for APIs as weel as for functional programming data structures.
Clojure(script) approach is a good example of how this can work beautifully well in practice. Java also used to be reasonably good in this regard. Javascript (the language) is good. Javascript (the ecosystem) is criminally bad.