In my specific case, I use an online-only bank. But I'm being petty with that objection, my bank still has enough information about me that I could verify my identity via a combination of a scanned ID card, proof of residence, etc... In general, point taken, that's a good distinction for most people.

However, substitute 'bank' out for any online-only service and the problem persists, because any reasonably smart attacker who SIM-swaps my account will still immediately change the phone number to point to themselves. If my Amazon account gets hacked, and the attacker goes into account recovery and changes the phone number, Amazon still needs a non-SMS based escape hatch for me to get the account back. The problem hasn't gone away.

So why not just use the non-SMS escape hatch all the time?

In Amazon's case, if they don't have your phone number they'll do account recovery over email. That's almost strictly safer, and just about as convenient as SMS.