One of the more common last names in Norway is still "Ødegaard", literally "Deserted farm", reflecting families who were residing on farms which still had not gotten a proper name by the time the tradition of using your fathers first-name as a last time started dying out. This even then - by the 1800's - involved re-clearing farms that had still not been re-settled after the Black Death.
Of the ca. 36500 farms which existed in Norway around year 1300, only about 16000 were still populated by 1520, and the population is believed by some sources to still have been only half of what it was in 1300. It was not until the 17th century that Norway "recovered" - that is, until it was back to pre-plague, levels, having lost several centuries of growth, but with population centres having changed, leaving many ancient farms still empty reminders.
In Norway, a disproportionate portion of the political and religious elite died. E.g. while almost the bishops in Sweden survived, all but the bishop of Oslo died in Norway. The Northern provinces of Norway were left to their own devices for a long time, without a functioning government, because so few government officials were left that the king chose to focus restoration in the South and largely just abandoned them.
That was one event.
I also don't recall running into Nazgul and Orcs while growing up near Oslo, and imagine facing that might have been a rather severe deterrent to growth.
EDIT: I'll add that Norway is a good reminder that a combination of destructive events, harsh conditions and the extremity of the hockey stick nature of population growth can explain a lot. Norway today has a population density of ~14 per km^2. In 1950 that would've been ~8. In the mid 18th century about ~3. In the aftermath of the plague, likely <1, and still <1 by the 16th century. You can still travel on foot for days through central Norway without meeting people if you stay clear of the places most popular with people out hiking for fun. A couple of centuries back, and not only would you be able to do that, but you'd have been likely to now and again come across farms that had been empty for centuries, despite centuries of a growth and recovery.
But you did have trolls, right? :)
It’s 1600 years of an active, organizing intelligence whittling the West away and which saw the _destruction_ of multiple major kingdoms and settlements prior to the beginning of the story. The people of Arnor were nearly exterminated as a result of the Witch-King, for example. These were not wars like medieval wars. They were wars that entailed abject slavery or genocide for the losers.