This argument actually does your argument a disservice. And I agree with your argument.
I've talked with plenty of NIMBYs in my neiborhood when an apartment building was going through approval. They aren't racists. And in fact they are often people of color. Their concerns aren't with poor people either. They tend to mention things like parking, traffic, density, character. Instead of immediately calling them racists, I think it's better to take them at their word and build and argument against it from there.
"I don't mind people being poor, so long as they don't drive on the same roads I do, and there aren't too many of them, and they don't in any way change the status quo as it existed the moment I finished unpacking my moving boxes."
Call it whatever you want, but a focus on property values is always a push for higher economic brackets, never lower.
There are empty lots around the corner from me, and I hope they build as much housing on them as possible.