Not only that, there is also diffusion caused by difference of concentration, which is increased by heat. And you have that concentration difference at the p-n boundaries/junctions in an IC.

Though I'm not sure how much actual damage you'd see in practice, whether the ICs tend to die with intense use before e.g. the capacitors mentioned above.