After that your efficiency declines until... Well you end up with fusion reactors that look like the sun. You also end up fusing all the mass that we need to live our lives on and make stuff out of.

If you don't think a civilisation could ever need the amount of energy that a Dyson sphere could provide, that's fine, we don't need one.

But if you imagine a civilisation that does need the energy a Dyson sphere can provide, then they make a lot of sense.