So if a manager says to the office, "Some of you people need to improve your performance" and one person takes that as meaning only people of color need improvement and reports the manager, you would side with the employee?

There are many common cases where someone refers to a group of people with the intention of meaning everyone while someone in the group interprets it as referring to a subset of the group. It stems from the ambiguity of the word people and the impact of tone and body language in conveying its ultimate meaning.