That's an assumption though, based on the current conditions and population. You can't extrapolate that same assumption to a future modernized world that has returned to one or two billion humans. Perhaps in that world, women happily choose to have more children than they are having now, for a variety of unforeseen reasons. Perhaps the world is just so great for everyone that women choose to bring additional children into the world, knowing just how much joy that children will experience. Perhaps children become a new popular hobby or some entertainment scheme we would find abhorrent in today's world.

I don't believe there's anyway to know today that it's a one way ratchet forever.