Hmm, it's more like any ionizing particle (charged) creates pairs electron/hole, many of them, so that the charge distribution in the pn junctions are altered. Sometimes gamma photons generate Compton electrons in the material that act as delta rays, to the same effect. Heavy particles can alter the crystal structure, inducing defects that may change its properties. Magnetic storage devices don't have polarised junctions and could withstand all this if they didn't include solid state electronics. Radiation hardening techniques and shielding can help a lot.
Its more a cause of the radiation ionizing the underlying semiconductor material. For instance when an electron tunnels into a transistor it can cause strange effects. Look up single event upset and single event latch-up for more information. The SEL is interesting as it causes a transistor to act as a double transistor and start drawing tons of power. At least that's how I understand it.
*I am not a physicist, someone correct me if I'm grossly wrong.