Yes when you can compile from scratch it does not matter anymore, but as you mentioned it has mostly to do with software vendors (OS distros, proprietary apps, etc). There are things that are basically disallowed for upgrades in the standard library because certain things (e.g. class member layout, default vs. empty user-defined constructor, such as the register promotion issue when pair<> was '=default' from the empty constructor).

I think Marshall Clow gives a pretty good overview of the ABI issues at a high level [0].

[0] https://www.youtube.com/watch?v=-XjUiLgJE2Y