Why not extend this to attractiveness, general health, enviromental factors like air pollution, gender, quality of local libraries, height etc., all likely influencing an individual's success in life?

Of course I don't think we should pick out elite based on these things either

No they should be picked for "ability", and it's all the same whether that "ability" was inculcated by good parenting or not.

The point is that two kids with equal ability can have dramatically different accomplishments based on things outside their control