Neutron radiation, plus the unfortunate geometric fact that the surface area/volume ratio of a fusion reactor will be low, compared to the fuel rod surface area/volume ratio in a fission reactor.

What this does is ensure that even operating right at the limits of neutron damage to the wall materials, the volumetric power density of a DT fusion reactor will suck. And that will destroy the economics.