Because, in general, there is more variance within groups than across groups, so you are generalizing that an individual person within a group is more talented / capable / whatever than an individual person from outside that group. Ergo, you are treating that second person "unfairly" due to his / her group membership or lack thereof.