Certain things are impossible with only 10,000. Others with 10,000,000. Still other things are impossible with 10 billion.
We want rich, diverse, dynamic and satisfying lives. We do not want a crowded, polluted, impoverished world. 10,000 of us is certainly too few to sustain our civilizational accomplishments. My guess is that the numbers we have now are too many. What is the right-ish number?
Ethically, we would need a pressure great enough to reduce the population for a horrible reason. We would never choose it.
We make policies all the time that affect peoples reproductive decisions. Many of these are, in fact, explicitly about biasing reproductive decisions in one way or another. For example, countries like Poland have introduced policies to increase fertility, monetary incentives to have more children. The purpose of that policy is quite explicit. Other policies discourage people from having more children, including improving access to safe and effective contraception, and either the withdrawal of fertility-increasing tax incentives or enabling people to postpone starting a family (which all by itself can slow or reverse population growth).
Yet, it seems that you only have moral qualms on one side of this coin. But perhaps I judge you too hastily.
That's because we've only been discussing one side of it. There are endless bad things that aren't worth denouncing in a particular conversation, if that conversation isn't about those things.