Banks have this in place already - EMV cards have powerful cryptoprocessors. In Germany we can use chipTAN, it's a small cheap reader for your card where you scan a six-binary-blinking screen that transmits the transaction data, then the card signs it and you get a six-digit TAN back. You can also manually enter the hash to be signed ("start code" is the technical term) and you get the TAN.

Customer support could ask you to authenticate using the TAN already, the hurdle is that you would need to carry the reader at all times.

Unrelated to banks, I believe it could be possible to extend SS7 signalling to not just transmit the caller ID but also a crypto signature/public key which the phone then can verify - or your phone provider could. Think of something like HSTS with a global database, if there is no match for the phone number the provider patches the call through, but if there is an entry, all providers can check for the public key transmitted by the caller and refuse to patch the call if it's missing or faked.