People are worried about the outcomes of such thinking, not so much the accuracy of it. Especially since it's only one step removed from talking about cultural/racial association.