Yep, it's almost more accurate to describe using phone numbers for 2FA as being anti-secure, not just insecure. That's because it's effectively no better than having no 2FA and it's possibly even harder to detect when your account has been compromised by a SIM swap. And many companies that use phone numbers for 2FA also allow resetting one's password via that phone number. It's really just a tragedy that companies do this, rather like when login screens prevent copy/paste.

If you're ever prompted to add a phone number to your account on some web service for "extra security", just click "remind me later" or "skip" as many times as possible.