Previously (centuries ago) most of the people were like the latter group. And things changed. I would not bet that a group can remain completely unchanged in the next tens to hundreds of years. There is of course a chance, but don't think it is as clear cut as you make it sound.
> Fertility rates also determine age ratios within a society
Plus immigration. Some like and accept that, some hate it and don't like it, we will see how it plays out, but there are more things involved than just fertility rates.
There simply are not enough people to replace this sort of decline, even if you just grab anybody regardless of skills, education, or even the ability to speak your language - which is not exactly a great recipe for a healthy country. Like if 60% of the world, by population, is losing 80% of their population over that timeframe, that'd be a global population decline of 48% with only 40% of people in the rest of the world. You'd need all of them and then a good chunk of their babies too, and that's for less than one human lifetime of change.
I'd say immigration is just kicking the can, but it scarcely even counts as that given how little of an effect it can have even at theoretically absurd levels.
Let me put it another way, if a pandemic would reduce the world population by 50%, what do you think birth rates would do?
Now, if your argument is "it will soon be a shitty time to be old and it will last for a while", with that I do fully agree. The old people of today have it much easier than the old people in 30 years. But people in 80 years might have enough (resources, space, etc.) to start making children again at different rates.
I would say we will live and see, but doubt it is the case for that long periods...
There's also the issue that I don't think I'm a snowflake. Fertility has become a big issue for me and is part of the reason I now have children - not as many as I'd like, but infinitely more than I was thinking about having not that long ago. I've come to view it as a core social responsibility since cultures/societies literally cannot exist without people having enough babies, and I think more people are coming to see it this way as well.
- If you dropped a human on a random spot on this planet, even if we just assume land, the median number of people within a mile would be exactly 0.
I think the second thought experiment is quite informative because it's easy to lose track of the reality when we mostly travel from packed city to packed city. But these cities, and the pathways that connect them, make up a negligible percent of the Earth. China is smaller than the US, and has pretty rough geography, yet was food self sufficient with a population upwards of a billion. They only became food dependent as their economy expanded and they replaced farms with iPhone factories, metaphorically speaking at least.
Finally, population growth in the longrun will always be determined by the groups with high fertility. It's never longrun stable unless everybody has the same fertility rate. So for instance the Amish today maintain a fertility rate of around 6. This is why there are now hundreds of thousands of them, and within a few decades they'll number in the millions. As highly educated, highly secular, high income individuals remove themselves from the gene pool, they'll simply be replaced by Amish. And as the former die off, and the latter grow - we'll eventually see population growth resume.
One last point. Fertility rates also determine age ratios within a society. Low fertility = most people are elderly, high fertility = most people are young. So it's not only economic collapse due to fewer people, but you create societies where there's a much greater social cost (due to elderly care), and far fewer people to take care of them. South Korea is a great example of this. If they don't fix their fertility rate, they'll collapse as a nation within the lifespan of some people alive today. 'Breaking News 2094 - North Korea successfully invades South Korea with one tank.'