Population on the whole will never become a liability. You DO want more young and middle-aged people than old so your tax base is healthy, but that's not a problem with population as a whole.

Finally, you always need a reserve of able-bodied humans who can be trained to become soldiers. Remember that a lack of trained bodies is one of the reasons why a nation like Japan lost.