It's more telling than you might realize that you took their complaint to mean "I'm upset I'm around so many white people" and not "I'm upset a part of the population isn't included in my area / culture". It certainly makes the rest of your comments less surprising.