It's naive and ignorant to ignore nurture in the debate. A lack of Vitamin D or cholesterol can inhibit myelin growth, which notably in the womb causes a miscarriage or still-birth/brain dead baby. It's reasonable to assume that a lack of Vitamin D or cholesterol in a child's formative years is going to cause severe developmental problems.
Natalie Portman has a degree and an oscar for one simple thing: She works hard.
If she didn't work hard she might be a recognizable actress, but she'd be like a huge number of university enrollees in that they never get any degree.
Genetics is a firm foundation, bad genetics can cause lots of problems. But only work is going to build a house, and only hard work is going to build you a mansion.
Anyway, your point gets into issues of free will and consciousness that philosophers have been debating for centuries. If everything has a cause, how can we say what we're actually responsible for, and how much is just a reaction to random chance? Perhaps consciousness is largely an illusion: the brain makes decisions based on firmly ingrained circuits that function at a subconscious level, and only later does that bubble up to the conscious level.
We may all end up near enough (for all practical purposes) at the same endpoint if we all worked hard and performed necessary rotations and frame changes given our situations. The brain is very malleable.
There is also the fact that people start at different places for different areas and that chooses where they end up specializing. And there is probably some self-fulfilling prophecy going on that trains people to avoid smarty pants subjects which are vital for developing rigorous thinking. why is it a matter of pride to be terrible at maths and technical subjects? or the fact of growing up with certain teachers and society all but directly informing you that certain races aren't up for certain more mathy subjects. I can certainly attest to that stuff messing up your ability to keep on when things get difficult.
We have nature, nurture and random chance that create all of us. I wouldn't have said "life's too short" and moved country if I hadn't read a single book (The Strange Life of Ivan Osokin) that I have no clue how it came to be in my possession.
My entire life is based on seemingly random chances. I met my wife via friends of friends of friends whom I originally met in a chat room. I got my dog (Dachshund x Jack Russell) by going to the SPCA when they were flooded with Labrador crosses and they had no terriers up for adoption, but she was getting taken out for a walk at that very moment.
Our entire lives are quantified by our reaction to random chance. I had less odds of being where I am today than I have of winning the lottery, but why do the events of my life occur so but my lottery ticket is always going to be a dud for the rest of my life.
It's the mixture of our nature and nurture that define who we are, by how we react to random occurrences. What random shit defined your life so far?
But more significantly, intelligence is a very fuzzy concept, much harder to describe and more so define than eye color or height. It is also as much of a cultural concept as things like beauty, charisma, etc... Which is also why it is a more touchy subject than eye color or height: it is much more related to one own ideology.
Because intelligence is much less understood than most of the rest of our bodies. For example, there have been various discoveries that seem to indicate our language has an effect on what we can know and understand [1, 2, etc.]. Perhaps there are dramatic differences in what each of our brains are capable at the "hardware" level, but the "software" (language) dampens the effect.
[1] The Piraha can't learn numbers after a certain age because they have no words to describe them
[2] An article a few months back about deaf people growing up without any language ended up bringing out the fact that some asian languages have words to describe some aspect of color that westerners do not. Babies are able to see the difference but gradually lose the ability as they grow up with no way of expressing it.
It's naive to think everyone is born equal that way.