The phenomenon Spandrell describes is a concern largely independent of whether the total population is increasing or decreasing.

Every advanced country has built a system for vacuuming up the most talented individuals from the countryside, and giving them important jobs in rat-race cities, where (for various reasons) they will have fewer kids. To counter this, you'd have to draw from somewhere else equally talented newcomers, and maybe a few countries can.