> Given the huge benefits of a Dyson Swarm and the fact that it seems to require no exotic materials and no more physics it looks increasingly likely that we are very alone in the Milky Way

There’s still a huge engineering gap between our current tech and the theoretical possibility of a Dyson Swarm. I don’t find it at all implausible that such a thing would need too much energy, effort, collective will, or some other resource, to make it practical. There could still be plenty of stealthy sub-Dyson civilizations, or even supra-Dyson civilizations that have found it in their interest not to be detectable.