If you have deals with many strangers do you not expect to get your hands dirty in some way?
This was something I changed when I realized that a tax related government agency's machine I was using about 10 years ago man in the middled SSL connections to my online banking. Who knows what was logged and what the retention on that was? I don't have time or the inclination to ensure every relative's / customer's devices are secure, so I use my own or go without.
In practice, it would mean replacing most of the world's user-facing computer infrastructure.
The card reader doesn't have to be any more tamper-resistant than the rest of the computer. On a public terminal, it should probably be built like an ATM's card slot (and the rest of the machine should be similarly armored), but on a personal computer it doesn't need to be any more robust than the keyboard a user would otherwise type passwords in with. The card-computer communication is all encrypted anyway, so even if it's built like any old USB peripheral it's still better than typing into a keyboard.