Yes, I see how direct conversion could work with alpha and beta radiation, but it seems the gamma and the neutrons would just blast through everything and you'd capture only a tiny percent of the energy.