society could fix all sorts of problems if we had a public key infrastructure...

Banks have this in place already - EMV cards have powerful cryptoprocessors. In Germany we can use chipTAN, it's a small cheap reader for your card where you scan a six-binary-blinking screen that transmits the transaction data, then the card signs it and you get a six-digit TAN back. You can also manually enter the hash to be signed ("start code" is the technical term) and you get the TAN.

Customer support could ask you to authenticate using the TAN already, the hurdle is that you would need to carry the reader at all times.

Unrelated to banks, I believe it could be possible to extend SS7 signalling to not just transmit the caller ID but also a crypto signature/public key which the phone then can verify - or your phone provider could. Think of something like HSTS with a global database, if there is no match for the phone number the provider patches the call through, but if there is an entry, all providers can check for the public key transmitted by the caller and refuse to patch the call if it's missing or faked.

Would you happen to know what kind of signature scheme they use?

IIRC the German system is proprietary, the specs are available only after payment of a couple hundred euros.

I am grossed out by proprietary protocols but proprietary encryption algorithms just make me laugh. Who even though that this would be a good idea? Are they seriously trusting their money with this?

I don't know about the German system, but here they use EMV-CAP: https://en.wikipedia.org/wiki/Chip_Authentication_Program

No public key crypto?

My bank seems to use a similar scheme. It appears akin to TOTP with 8 numbers. But the secret is inside the black box. They also have something like a QR code but with RGB colors (does not work with blue light reducing features).

We are in some kind of Stone Age of The digital age...

Or maybe some sort of interconnected web of people who trust each other...

That would be pretty good privacy.

Like the web of trust from GPG?

The WoT originated with PGP (though obviously GPG implemented it as well), but yes, that was the joke.