I think another factor is traditionalist religious minorities – ultra-Orthodox Jews, the Amish, Latin Mass Catholics, etc – many of these minorities still have big families, and are sustaining those big families generation after generation. (Even though some people who grow up in those kinds of communities defect to mainstream society, something like 80% choose to stay.)

Those religious minorities are quite small right now, but if they manage to sustain their current growth into the coming decades, they may end up being a much more substantial component of the population.