Having 2 children to replace the parents is already enough to drive population down longterm (because statistically, there could be a lot of reasons that those kids might not reach a time to have their own offsprings).