There's always a tradeoff between convenience and security, but I'm relatively unworried about someone, say, breaking into my bank compared to someone spoofing my phone number.