As another commenter suggested, there may also be signal processing techniques to "mod out" the interference from the music (perhaps measured with a probe on the audio output?) and measure only what's left.
Better silicon should solve some of these issues, for example adding some additional isolation between the power section and the logic a small super-cap with decent in-band filtering might do the trick.
As far as radiating EM goes I'm not sure what can be done but some more additional shielding and EM noise reduction should add some degree of protection.
Most implementations will eventually be vulnerable to some type of side channel attacks, the complexity and cost-vs-benefit is important here.
Launching an evil USB attack on the off chance of getting a key is most likely not very scalable, but considering that NFC/wireless payments for phones will become more and more common, and that crypto-currencies might actually end up being in common day use being able to extract keys during signing from EM leakage might just be the natural evolution of ATM/Payment Cards skimmer attacks.
In the past 2-3 years we had various key extraction attacks using "strange" vectors like EM/audio extraction, temperature, cpu usage, cross platform cache attacks, these attacks can threaten cloud computing and mobile computing quite severely unless we can root them out and being to modify the devices and platforms to be much more resistant to them.
Can you impede the extraction process simply by having your phone/tablet/laptop generate interference while the decryption process is happening? For instance, if your laptop is simultaneously playing music through its speakers, would the analysis still let them definitively pick out the signing key?