Men don’t even exist until the Second Age, and “cities” at that time are basically just family settlements, with maybe a few dozen generations of men. And at first they’re below replacement rate. Look at the family tree of Beor: http://tolkiengateway.net/wiki/B%C3%ABor#Genealogy… no one is having three kids. It’s two or one.
The Third Age is only a few thousand years later. There’s just not enough time at those kind of birth rates to blanket a continent the way humans have.
It's also plausible that societal advancement was also just slower than with us when the author seems to assume it would be expected to advance at the same pace. (i.e. it's taken ME longer to reach a medieval level of advancement.) Certainly happened with plenty of areas in our world for geographical determination and other reasons--though, by effectively modeling ME on Europe and giving it horses and AFAIK familiar crops, that's a harder case to make.
Unlike the humans from a certain other famous series, we knew the background of Gondorians all the way to their earliest fathers around year... 300? 800? Of the first age of the Sun.
Now, discounting that... I agree, ME has really low population growth. Might be caused by Gondorians and Arnorians living to 200+, 300+ years old earlier, when their culture were established; when the blood grows thin and they live like normal humans, their cultures ossified already; 40 - 50 year old normal human grooms and brides aren't as productive as 80-100 year old Numenorian grooms and brides.
Maybe the women are missing from not just from social life, but from the world. Not just the entwives[0] and the rarely seen dwarven women, but even human women are surprisingly rare. How many of all those many men making an appearance were married? Only hobbits seem to have actual families, but still not very big ones.
Maybe Middle Earth just isn't very fertile and can't sustain large populations. Maybe there's a very good reason why population levels are closer to Siberia.
But it also seems that Frodo and Bilbo were NOT mentioned as being "strange" or anything, so it's not certain that everyone would marry and have a family of any size.