We also know that although secular changes in diet can shift population averages over time, the heritability of height is pretty high: most estimates end up at about 0.8. If your parents were short, getting adopted into a tall family is not going to help you become taller.
Height is a highly polygenic trait and no single gene controls more than a few percent of the variance. When people say that we don't know the "genes for height", that's what they mean, not that the jury is still out on whether height has any genetic basis at all.
Years ago, I read an article in a magazine while waiting to be seen by a doctor. It talked about a study in a culture that valued boy children over girls.
Boys, especially oldest boys, got treated consistently better than girls. They were more likely to be fed their favorite foods. If they didn't feel well, they saw a doctor now whereas there was more often a "wait and see attitude" with girls. This meant boys were more likely to see a doctor several hours sooner, such as in the evening rather than waiting until the following morning.
This was not a case of girls being treated abusively. It was just a case of boys being somewhat pampered.
The difference in outcomes could be measured in terms of mortality rates.
We don't know what all genes contribute to height difference. We do know that underfeeding people will stunt them. Girls around the world seem to constantly be on a diet for fear of being too fat.
We don't know how much female height is being suppressed by cultural factors and we also don't know if cultural factors suppressing height might have intergenerational impact, even influencing genes by gender.
I don't think it's unreasonable to wonder about such things. I know for a fact it is enormously hard to have a meaningful discussion about such things on the internet, no matter how scientific the audience in question imagines they are. People are quick to dismiss such suggestions.
I've seen ridiculously biased statements about gender on HN upvoted to the top position in a discussion and heartily agreed with. It rapidly becomes a hostile environment where it doesn't feel safe or in any way productive to point out glaring flaws in such remarks.
Perhaps you could get some women to a similar level as men by giving them hormones from an early age (I don't know), but even then I think it would be a stretch to say "see, it is all cultural, we are denying those girls their hormones".
I've taken some pertinent college classes, like Intro to Psychology and Social Psychology. These are really hard things to debate in good faith because people look around them at the world, see the reality that men are, on average, taller than women on average and assume genetics when we don't actually know the reason.
So if you have good faith reasons to wonder put loud or speculate about such things, you are likely to get treated dismissively and not taken seriously.
There is a huge component of "them that has, gets" and it's really hard to sort out exactly why x person got y outcome. If you are interested in social justice -- or even just making your own life work when you are the "wrong" demographic -- then these are really frustrating discussions to try to have. It may be some hypothetical argument for someone whose life works well, but it's often a desperate attempt to get a fair shake of some sort by marginalized people.
So there's a lot to unpack there and it's hard to unpack it and there are legitimate reasons why some people feel strongly compelled to not go along with "the obvious answer" that "everyone" knows to be true, even though they may not know how to effectively argue against it.