Well, right. You could have a distribution of black holes. But that would mean a lot of black holes - all of them in space so void of other matter that they have no accretion disks. Also, with black holes so common, we would see many collisions with plain stars - and those would be very visible.
Doesn't need to be centralized, no? They can orbit the galaxy as the other bodies.