I think you've hit on one of the main reasons people get uncomfortable talking about genetic explanations for behavioral differences; it starts to sound like you're saying that some groups can't be helped or that there's an implied ranking of populations. Neither, of course, follows.