"Given the huge benefits of a Dyson Swarm and the fact that it seems to require no exotic materials and no more physics"

Since we haven't been able to build such a thing I think it's fair to say it requires something or other that's exotic to us.