Due to redlining, black people generally couldn't buy houses. They rented. Having multi-family units means letting renters into your neighborhood, which means letting black people in. In many part of the US, politicians in county elections often run on "eliminating multi-tenant housing to reduce crime", which is a dog-whistle to racists.

I don't think it's necessary to provide references here. The history of housing discrimination is well researched and just a quick google search away for you. I know this is HN and the culture is to require proof, but seriously, some people just need to educate themselves, because otherwise they aren't receptive to listening.