For this to match observations, all galaxies visible outside the milky way would have to be Dyson sphered through their whole extent. If they weren't, it would be obviously visible, as an expanding civilization that had only spread throughout half a galaxy would be a totally lopsided galaxy. And for some reason, all of those civilizations must have decided to leave a certain percentage of stars un-sphered even though they have the capacity to sphere them all.

And the Milky way would have to be un-dyson sphered, because if it were, we would be able to notice these spheres close up. e.g. if dark matter is 5x regular matter, we would expect there to be 5 Dyson spheres of stars the size of the alpha centauri system all within 4ly of Earth. This would be easy to detect through occlusion. Also, gravitational interactions between sphered stars and unsphered stars would be easily observable in our own galaxy. So for this to work, at minimum, the whole observable universe must by Dyson sphered, except for our own galaxy.