Leakage current usually increases superlinearly with voltage, though. If the tantalum's DC leakage is 10 μA at 50 volts, it's probably closer to 0.1 μA at 5 volts.
In capacitors with a given dielectric, the maximum energy is proportional to the volume of the dielectric, so although a 47 μF cap charged to 108 volts stores the same energy as a 220 μF cap charged to 50 volts, it also needs the same volume.
(I'm not an EE either; I only play one on HN.)
https://en.wikipedia.org/wiki/Human_eye says an adult human eyeball is about 23.7 mm × 24.2 mm × 23.4 mm, and a spheroid like that is π/6 of the corresponding cuboid, so you only have about 7.03 cc to work with.
So probably you'd have to settle for under 1 cc of capacitor, which (in the case of X7R anyway) means storing more like 15 seconds of 2 mW.
How much does that weigh? How big is it? What's the leakage current?
If it takes enough space away from the betavoltaic device, and thus reduces the power generation, then the cap will never fully charge.
Not knowing what you're thinking of, I picked https://www.mouser.com/datasheet/2/427/134d-1763800.pdf mostly arbitrarily. Note that I'm a programmer, not a hardware person.
The lightest weights 2.6 g, which is 1/2 the weight of the eyeball, so cuts the power generation to 1μW.
I think it's telling me there's a 10μA leakage current, which I think means the cap will power up to 0.1V before the leakage rate matches the power rate.
Now that you know my level of ignorance, what's your take?