SMS isn't resilient to the worker at the local retail store for the phone carrier giving someone else a SIM for my phone number. That's a much bigger threat vector than Google/iCloud/a sync target I manage storing an encrypted version of the TOTP credentials.

How realistic is this threat? I would think that the employees would have to jump through hoops that require you to be present (or at least a lot more of your info to be stolen than just your name and number) and that the home network would detect a duplicate E.164 number with conflicting IMEI/IMSI numbers and locations pretty quickly.

FWIW: https://en.wikipedia.org/wiki/SIM_swap_scam

This is more like confused deputy than collusion (though that can happen as well), but nevertheless the end result is somebody else ends up with your number, and your device gets deactivated.

If I lose my phone I can go to the office of my carrier, present my ID and receive a new SIM with the old number[0]. If Apple/Google decide what I'm not their customer anymore then I have literally zero ways to recover anything from them.

[0] and half a year later the bank would finally found out about and block the SIM 'to prevent fraud' at the most inconvenient time. But again, it's solvable with a visit to the office and an ID.