EDIT: As a non-biologist, but someone who can't read enough about science, I recommend the writings of either Richard Dawkins or Stephen Jay Gould. Both, actually, since they disagree on a number of points.
That said, I want to clarify something. When I speak of evolutionary fitness it is not a judgement of the individual regarding their actual value to the species or society. Evolutionary fitness is, really, a cold thing. It only matters whether the individual/species reproduces/survives. The dodo was an evolutionary dead end, even though the cause of extinction was humans, a pressure was applied that the species could not overcome. An individual that does not reproduce is another dead end. We, as a society, may look upon a Harrison Bergeron as the epitome of fitness (mental, physical), but if he's shot down before he can reproduce evolution doesn't care about him, his fitness is nil.
Is there any reason to believe that genetic issues obey a distribution where they are either very extreme, or have no effect at all? i.e. if everyone not classifiable as having a really extreme disease is equal? I don't think so.
I think it's much more likely that there is a normal distribution of breakdowns in fitness - a few very severe problems which manage to survive to birth, then many, many more variants with negative fitness, or ones which are good but very picky, and which may be left over from environments where they were +fitness. Then there are probably variants which are all-around good.
So the idea that we're all born with the same fitness level is a pretty extreme example of invalid "blank slate" type thinking.
Anecdotally, I grew up in a poor rural area, and of my gifted class (of kids with 130+ IQ), 15% didn't even finish high school. And, I'll estimate that 50% didn't go to college. In fact, of my immediate family (6), all have been gauged as having high IQ. However, I'm the only one that had finished college without help (by 90% luck). And, a couple of my other siblings have now finished college with my financial help. In my experience, location and family are much more important to potential achievement.