Sure, you're totally right that in some cases, physically losing a phone also means loss of that phone number. But in 100% of cases, physically losing a phone means loss of those TOTP tokens.

For the vast majority of people, that's a dealbreaker, in spite of SMS being deeply flawed, as the comment you originally replied to said.