But, yeah, it's all because they have brown skin. Oh, wait, the tenants in question are whiter than I am.
If you want to know why we don't bother, it's because the owners of the surrounding houses use our place as dumping grounds and the owners have been there for decades and know all the housing inspectors and police. It seems pointless to even bother when the landlord wants the house to fall into disrepair and the neighbors will use their connections to authorities to dump on our grounds again as soon as we clean it up. On one occasion I was sent a notice that I would be imprisoned because of trash my neighbor put somewhere not even on our property and before I even moved in. I dutifully cleaned it up because it was clear I was going to be framed, but as that took the entire weekend I had little energy leftover afterwards to do yard work around my own place after being exhausted dealing with the neighbor owners.
Meanwhile the neighbors' house indeed did look spotless because they dumped all their trash on the renters' yards.
We know how people treat rental cars. I don't know why it's a big shock that people will treat rental properties in the same manner, and why other folks would rather not live next to that. Blame race and class if you like, but to not allow at least a little more nuance to the discussion is silly.
If people are going to do restrictions about who lives where, HOA's are probably better than zoning. They're more localized, and the costs of administering them are born by those who perceive them to be beneficial.
I'd never live in one, but to each their own. I'd rather have someone choose one of those in a free market than get involved to zone the entire city one way.
At least in my area, it's all about people wanting the area to remain semi-rural. They generally aren't opposing low density SFH in independent builds. The people in my area are against many development and medium density homes.