You could nominate a number or bunch of numbers that you trust (possibly with the caveat of them having to be with the same carrier), any of which could validate a request for a new SIM. Valid events would likely be a rare occurrence and so unlikely to annoy your family and friends much, but might significantly increase the difficulty of SIM-swap attacks.

Or have a password that you set up for this circumstance. If you forget your password, your password reset information is mailed to your billing address.

My bank once had an "IRL password" like this, which I'd use when calling them. They silently dropped this feature after a few months. I'm guessing it results in negative ROI once you factor in the number of people who would opt-in despite routinely using password reset as their login method.

That password thing isn't as useful as I'd hoped. Mine is maybe 20 digits and I use it frequently when I'm calling my phone carrier for routine activities - they do check it.

For traveling overseas I needed a different SIM card and bought one at a local store (to be repaid by my carrier). The clerk at the store did not need my password because I showed him my valid identification. Password was useless in this case.

Not counting fake IDs that get past clerks, a few years ago Krebs On Security noted that the normal bribe at a phone kiosk was $85 to do the SIM swap illegally.

Even without a bribe, you still have to trust the minimum wage worker at the store. Motivated criminals or organized gangs could send people to apply to jobs at mobile providers. Come to think of it, that's probably already happening... spies too...

There's a big gap between the high trust and low salary that companies give their frontline employees. And despite (or because of) that gap, they're not motivated to do extensive background checks.

But the average employee at the Verizon kiosk in the local mall is arguably handling more sensitive data than an employee at the passport office in the State Department.

In this case the phone was presumably destroyed, so a reply would debunk it, a call would debunk it instantly by the fact of a connection to the phone. If it's stolen, the thief can be asked to confirm their identity in person.