Asymmetrical distributions don't fare well, in part because the mean probably isn't the thing you want to be comparing anyway. Mixture distributions or data contaminated with outliers are also trouble.
On the other hand, a bunch of simulation studies have found that t-tests work pretty well for symmetric distributions, even those with moderately heavy tails like a t(6) distribution. Moreover, the power declines a lot faster than the Type I error rate (which, for whatever reason, people seem to worry about more).
I've gotten a few paper reviews where someone has made the "but it's not normal" argument. Redoing the analysis with permutation or randomization tests (or both, for one stubborn reviewer) almost never changes the resulting p-values by more than a few percent. I don't know if this reflects the fact that they're usually quite similar or that we're just fairly judicious in how we analyze data.
For cases where one needs too test "is this bigger than that ?" I test for stochastic dominance rather than equality of mean.