If you only use RSA to sign and not encrypt, then you lose the padding oracle attach and you're left with... a ton of other issues still. But the Bleichenbacher family of attacks is really devastating, so I'd tolerate some RSA digital signature usage. But please, stop using RSA key transport!!