To fix the problem, we either have to stop referring to any metaphor/symbol involving color with negative connotations; or we have to stop using color to identify and refer to people. I think the former is good for precision (allowlist/denylist are great identifiers in that regard), but won't really solve our other problems; while the latter is probably better for human dignity, mutual respect, and combating our propensity for tribalism/racism. (Or, why not, we could do both.)