But sim swapping is more a targeted attack. There are probably several orders of magnitude more opportunistic attacks than targeted attacks.

If you can make opportunistic attacks fail and require targeted attacks, you still have a big advantage.