Most likely answer, religion. Successful religions grow one of two ways, and generally have encodings for both. That is they tend to have means to convert the non-religious, and secondly, and importantly in this case, have 'strong' rules on having kids. For example, in the old testament Onan was killed for 'spilling his seed' and not impregnating his dead brothers wife.

That said, even in ancient times it appeared that birthrates did go down, as in Greces case, their city state succeeded. This left them open from attack and cultural take over by outside nations and ideals with higher birthrates.

In the modern world we are coming to the point where human labor is not as necessary and has been supplanted by machine labor. Also machine protection, in our weapons of war. This could allow a nation/culture to shrink dramatically while still being able to defend against outside threats. If such a shrinking culture were one that 'grew' by spreading its message to other cultures it may succede to grow and shrink the entire worlds population at the same time. These ideas of disappearing utopia cultures are often touched by science fiction.