> To me, he’s making a sacrifice for me and the rest of the team, and I think it hurt morale, even though he probably intended the opposite

This is the wrong take. It's a sign I believe of a junior or a team who is not confident in their output.

If the manager has to leave before you then by your rationale, you're promoting a lax top-down attitude, whereby the manager doesn't seem interested.

> How am I not supposed to feel guilty, even if it’s his choice?

By doing your own job and letting him do his.