That's actually a good question and has to do with the physics of respiration. When you inhale, you're creating negative pressure that pulls particles closest to you into your lungs, leaving a void that other particles fill as they randomly redistribute themselves. This causes aerosolized droplets to follow the path of least resistance around the mask, as if they are pulled, rather than landing against it. That's why real N95/100 masks require a proper fit: if there's even a sliver of space between the mask and face, inhaling will simply suck air in through that space instead of through the filter, which has a higher "resistance".

When you exhale, on the other hand, you create positive pressure which violently expels the droplets regardless of what's in the way. This creates droplets with high velocity and turbulent flows within the mask that prevent them from traveling a straight path through the mask and cause most of them to collide with the fabric.

Basically, once it's exhaled it's a hazard to anyone who doesn't have a properly fitted N95/100 mask (which requires specialized equipment, though it's relatively common)