I think the issue is the low absorption rate. Eli Yablonovitch proposed a box of PV cells facing inward containing a heat source where the IR light would bounce around until converted to electricity or absorbed as heat. This could be used inside a water heater so waste heat could be stored. Known as thermo-photovoltaics. See this talk I think: https://www.youtube.com/watch?v=lDxJsa8miNQ

Yes, I see how direct conversion could work with alpha and beta radiation, but it seems the gamma and the neutrons would just blast through everything and you'd capture only a tiny percent of the energy.

Sorry, I was responding to the part about IR.

You have to go the other way of the spectrum toward something that loves to avoid interacting with matter...

Finally a use for dark matter

But dark matter is matter that doesn't react very much with radiation (including light), so it would react even less.

We just need some blimp attached mini suns floating in the sky in the night and this can power all the existing solar panels on the ground.