If you have the technical know-how to protect your private key, and you're immune from rubber hose attacks, and there's no mechanism for key recovery outside your control, I'd agree.
However, people.
And I'm not aware of any system suitable to replace SSN that can withstand rubber hose attacks. That's where law enforcement & the justice department will have to step in, as always.
https://arstechnica.com/tech-policy/2015/10/how-a-criminal-r...
If your card holds a private key and won’t sign anything without also being fed a correct PIN you’d need a proper exploit of the application on the card to defeat it.
Like - US for a very long while didn't use chip & PIN - but it was simple to deny the transactions and as a consumer you were somewhat well protected, despite lax security. But this generates large costs, and perversely, the cost/uncertainty affects the small businesses the most.
Yes, when I use 3DSecure for an online transaction, the bank shifts the burden of proof to me (a correctly-authorized 2-factor transaction cannot be simply challenged with a complaint at the bank, I have to prove that it wasn't me). Still, it's hard to argue that 2-factor for online transactions is consumer-unfriendly, even if the consumer loses the deniability
By that I mean that we need to educate the masses that a few letters and numbers are as valuable and powerful as the combination to their home safe or keys to their car. People just aren't inclined to exercise the same degree of caution over some gibberish letters as they are the cash in their sock drawer, even though the gibberish is potentially much more valuable.
I'm skeptical that such an education is possible without it being ingrained from youth and even then it may be impossible to achieve a high enough degree of understanding to make the kind of shift you envision.
If people have to go through a lot of trouble, I suspect they would quickly learn a way to minimize the risk of having to do so. I suspect that most people are careful with their passports so that they don't lose them, because it is a bit of a hassle to get a replacement.
1. The use of a physical authentication mechanism makes exfiltration far more difficult. The problem with passwords is that they can be sniffed with absolutely no awareness by the victim. A device or token you have to have yourself is going to make its absence known.
(This presumes it's not cloneable.)
2. That reduces the attack surface for phishing. What I suspect is is that you'd see other forms of attack -- say, trying to get recurring charges inserted into bills or some such, rather than full account access.
3. In the event of token compromise, the ability to determine after the fact that a device was compromised and invalidate the subsequent transactions would be helpful. (Sorting out who bears the risk for that would be an interesting question. I'd like to put the burden on the banks, as they're best positioned to detect such activity.)
Gibberish is not something humans can remember. Technical people use a password manage, but that just limits us to one password that we use often enough that we have a chance to remember it. One wrong knock to the head though and all the passwords stored in the manager are gone - unless you wrote the password down someplace, but then that place is your sock drawer: probably not secure enough.
SSNs are lost for good on breaches of companies' infrastructures, but nothing should be lost in a Pub/Priv key scheme. What we'll need is a good key revocation process. So when my identity is compromised (say, malware on a device) then I can be me again.