IIRC, Authy uses this SMS challenge to authenticate users to allow you to download the database of secrets. This db is encrypted by a passphrase which never leaves the device. So, Authy is using SMS as one part of a multi-auth system to get to the final unencrypted data.

That said, anyone with a SIM card able to get messages for your phone number can download the encrypted database and attempt offline passphrase recovery. It might take a while to brute force, they seem to use a lot of hashing rounds as it takes a couple of seconds to verify the passphrase on a 2014 Moto X.