It actually has nothing to do with the mirror. It’s because when you rotate the image to align its ‘front’ direction to your body’s, you naturally perform this rotation about the vertical axis. It is this rotation which has the effect of swapping left and right.
(It may also help here to think of a mirror on the ceiling. Consider: does this also swap left/right, or does it swap up/down instead?)
However I orient myself and the mirror, it feels like there is something special about my left and right according to my up and down. Why would I naturally rotate the object about the vertical axis to align it?
Easy way to consider it, when you look at another human being, you innately understand that their right is your left, so you're rotating your perception of their directions around the vertical axis. The mirror isn't actually flipping anything, the top reflects back from the top, the right reflects back from the right. The "flip" is your brain treating them as a person rotated 180°, not the front of a person as if viewed from behind
EDIT: I understand that it's front and back that actually get reversed, but that still doesn't help my understand why we think that left and right are reversed by don't think up and down are reversed