Even if you are for depopulation, I think you would still like for the smaller population to be well fed, educated, and socialized. Those people would need to support the older and larger population both economically and in other ways.

The fairness argument is just a general argument against all redistribution of wealth, if you are against all redistribution then many popular programs and policies are already failing your moral test.