It only seems that way because pre-1960, the world was entirely dominated by sects (e.g. religions) with large families. All the old religions are very traditionally pro-natalist for a reason. Go forth and multiply!
If the population didn't grow, it was only because there was just no more food to feed more mouths. So it was for all of human history until recently.
It's only now that there is a significant group of people who choose not to have large families, and who have the birth control technology to make it happen. As with past sects who followed this path, they will dwindle as a proportion of the population. We are in nothing like a steady state, and high-fertility groups absolutely will own the future, just as they owned the past.
Topical here is The Most Important Graph In The World:
https://www.unzcloud.com/wp-content/uploads/2015/09/Populati...
Also, Orthodox Jews have been around for a long, long time and are still a small minority almost everywhere. Small ideological sects with large families historically don't seem to take over the world.