Men don’t even exist until the Second Age, and “cities” at that time are basically just family settlements, with maybe a few dozen generations of men. And at first they’re below replacement rate. Look at the family tree of Beor: http://tolkiengateway.net/wiki/B%C3%ABor#Genealogy… no one is having three kids. It’s two or one.

The Third Age is only a few thousand years later. There’s just not enough time at those kind of birth rates to blanket a continent the way humans have.

It's also plausible that societal advancement was also just slower than with us when the author seems to assume it would be expected to advance at the same pace. (i.e. it's taken ME longer to reach a medieval level of advancement.) Certainly happened with plenty of areas in our world for geographical determination and other reasons--though, by effectively modeling ME on Europe and giving it horses and AFAIK familiar crops, that's a harder case to make.

Unlike the humans from a certain other famous series, we knew the background of Gondorians all the way to their earliest fathers around year... 300? 800? Of the first age of the Sun.

Now, discounting that... I agree, ME has really low population growth. Might be caused by Gondorians and Arnorians living to 200+, 300+ years old earlier, when their culture were established; when the blood grows thin and they live like normal humans, their cultures ossified already; 40 - 50 year old normal human grooms and brides aren't as productive as 80-100 year old Numenorian grooms and brides.