It doesn't completely annihilate to produce pure energy. It produces helium-4 and two protons. Or you can react helium-3 + deuterium to produce helium-4 and one proton. The point is that helium-4 and protons are easier to shield against than neutrons, don't turn your reactor radioactive, and at least in theory their energy can be extracted directly (eg through induction) instead of through heat.

Edited to add: except helium-3 + deuterium still produces neutrons, because sometimes the deuterium will react with itself to produce helium-3 and a neutron.