If you have deals with many strangers do you not expect to get your hands dirty in some way?
This was something I changed when I realized that a tax related government agency's machine I was using about 10 years ago man in the middled SSL connections to my online banking. Who knows what was logged and what the retention on that was? I don't have time or the inclination to ensure every relative's / customer's devices are secure, so I use my own or go without.
In practice, it would mean replacing most of the world's user-facing computer infrastructure.
The card reader doesn't have to be any more tamper-resistant than the rest of the computer. On a public terminal, it should probably be built like an ATM's card slot (and the rest of the machine should be similarly armored), but on a personal computer it doesn't need to be any more robust than the keyboard a user would otherwise type passwords in with. The card-computer communication is all encrypted anyway, so even if it's built like any old USB peripheral it's still better than typing into a keyboard.
The first makes your public key useless for security. The second locks you out of everything.
So you've still got to have patterns-of-fraudulent-use detection, and some sort of token-reset mechanism.
Persistent data encryption is another option, but that compounds the key-loss problem in that you cannot access earlier data.
I have an encrypted file with website|username|password on my laptop to allow for unique passwords across sites. But it's a pain, and definitely not something everyone will do.