The main issue with SMS TOTP seems to be that phone numbers can be trivially stolen via social engineering, or less trivially via direct cell attacks. A TOTP app is effectively immune to both of these attacks. All your concerns are still quite valid however.