Depends on the timeframe you're looking at. Within a single generation you're correct.

Once you step into more generations though, fertility rates drop as child survivability rises, as it becomes much more viable to rely on a few children as insurance for old age. Source: Hans Rosling, Gapminder