Better silicon should solve some of these issues, for example adding some additional isolation between the power section and the logic a small super-cap with decent in-band filtering might do the trick.
As far as radiating EM goes I'm not sure what can be done but some more additional shielding and EM noise reduction should add some degree of protection.
Most implementations will eventually be vulnerable to some type of side channel attacks, the complexity and cost-vs-benefit is important here.
Launching an evil USB attack on the off chance of getting a key is most likely not very scalable, but considering that NFC/wireless payments for phones will become more and more common, and that crypto-currencies might actually end up being in common day use being able to extract keys during signing from EM leakage might just be the natural evolution of ATM/Payment Cards skimmer attacks.
In the past 2-3 years we had various key extraction attacks using "strange" vectors like EM/audio extraction, temperature, cpu usage, cross platform cache attacks, these attacks can threaten cloud computing and mobile computing quite severely unless we can root them out and being to modify the devices and platforms to be much more resistant to them.