"Doing a half-day?" is the remark I hate hearing the most when leaving at 5. It's said as a joke and yet it's not.

Upvotes for the person who can come up with a good come-back!

(Note, this is not a problem where I work at the moment. And even at places where it's been said, I just ignore it. Just leave at 5 people!)