My knowledge of convert channel attacks and timing attacks mostly dates to the 90s. Here's what little I know, for whatever it might be worth:
1. If your crypto primitives take variable time and your attacker can see that, you're hosed. Any encryption system with variable time encryption is useless unless you can somehow quantize it safely.
2. Eliminating covert channels is very nearly impossible. The literature on this goes way back, and it's ugly.
Should most users defend against these attacks? The timing attacks, yes. But I would assert that if a hardware encryption module appears to operate in constant time, most users should trust it. But as always, it depends on your threat model.
For something like a serious credit card vault, I still think your best bet is to get it off the network, build and audit a custom protocol that only passes a known set of messages, rate limit those messages, and quantize message timings. At that point, your biggest problems are bribery, insiders and physical security.
But if you do all that, I can't imagine why you would want to generate your encryption keys using somebody else's black-box hardware. Entropy pools are a good idea, even if you decide to trust your hardware AES implementation.
(And yes, I'm still oversimplifying. Hacker News is no substitute for a real threat analysis and security audit, nor for a conversation with whoever regulates or insures your business.)
For this threat, most active attacks are not particularly useful. They may capture encrypted data at a different point in transit than the one that originated it, making most side channel attacks difficult or impossible.
But a compromised RNG can be almost impossible to detect, and yet render almost any crypto system trivially breakable offline after the fact. The NSA has already once created a DRBG for which there could be a secret key that could determine the internal state of the generator with only 32 bytes of its output http://www.schneier.com/essay-198.html
I agree that being too wary of your hardware can lead you down a rabbit hole of paranoia, and that on the whole, we have much bigger problems to solve than hypothetical backdoored hardware random number generators (I have not seen any evidence that Intel has included a backdoor, merely people saying that because it's a black box we can't tell that they haven't). But since a good RNG is absolutely essential to so much crypto, and a compromised RNG would make it so easy for an adversary who passively monitors all communications and would like to have the option to decrypt them at their leave at a later date, I think that this is more worth worrying about than problems with the AES instructions.
Have you read any of these papers? I respectfully guess that you haven't. You're in luck. Go read Boneh's "Remote Timing Attacks Are Practical", and Osvik & Tromer's local spy process paper.