Generally, these people are not making sacrifice. Some take time off at different times (come in later, don't come some days). Some are intentionally stretching work so that they are not at home (it feels better then dealing with toddler, the partner is jerk and they did not crossed to divorce, they are lonely and this let's them forget). Some need to feel important. Some actually watch youtube a lot during day (found out from reflection in the window).

Whatever it is, they have reasons that have nothing to do with me. If I slack with work, I am in the wrong. But if I work in full speed, it no reason for guilt.

This is the wrong take. It's a sign I believe of a junior or a team who is not confident in their output.

If the manager has to leave before you then by your rationale, you're promoting a lax top-down attitude, whereby the manager doesn't seem interested.

> How am I not supposed to feel guilty, even if it’s his choice?

By doing your own job and letting him do his.