People downplay the probability and importance of these issues, but the situations where you loose your phone are often the situations where you need access to your online accounts. One such situation can do far more damage than all the hacks combined. (For example, someone steals your backpack with your phone and wallet. Horror scenario: someone steals your backpack with your phone and wallet in a foreign country.)
In short, loss of access must be considered as a tradeoff.
Loosing anonymity due to phone-based 2FA is another issue that never seems to be considered in these discussions.
Finally, phone-based 2FA discourages you from splitting your accounts. (It's a bit of a hassle even with one email. Imagine managing 5 or 6.)
Not sure how anonymity factors in here. How can typing a number into your phone to set up TOTP then typing the resulting numbers back into your already-authenticated account deanonymize you?
Finally, yes, adding more security is a hassle. But if you're willing to add 5 seconds to your login on one account, I don't see why you'd be super against doing that for multiple accounts.