TL;DR: it's more about demand than labor supply. Same result.

I don't think it is as binary as you suggest. There will be lower total saving after a delay. There will certainly be less investment more immediately.

What will happen is this.

If you borrow money to make and sell (say) shoes, and the population is doubling every 35 years (2% growth), then your market doubles in 35 years also. (You need money because your buildings and equipment wear out, if not for materials.)

Your lenders have it relatively easy in this situation, because fewer of their borrowers go bust, so interest rates (which are partly compensation for risk) are lower.

With negative population growth, your market is shrinking, so your lenders' risk of loss increases.

Anyone who still has excess savings demands higher return on investment. Fewer potential investments can jump the higher hurdle. Total investment decreases, total income decreases and everything spirals down.