Not that I recommend SMS, but if thieves steal my phone I turn off its Google voice connection and it stops getting text that particular phone #, which I can still get just fine in Google Voice. More to the point I do tend to feel like N-factor auth is often a sort of Matryoshka doll thing on one end or another more than anything.