If the mask becomes contaminated, it becomes an ideal vector for encouraging infection -- so if you touch the inside of the mask with your contaminated hands (say you move the mask to touch your face or adjust it or whatever), for example, the mask becomes the opposite of protective.
Masks are somewhat effective at reducing spread if the already-infected person is wearing the mask.
Also my understanding is that a N95 mask that isn't properly worn (doesn't have a good seal) then it's nearly equivalent to no mask at all.
So you combine those two together and you have people who aren't effectively protected by the mask and are making riskier decisions.
--
I have looked at a fair bit of research on masks lately, and everything I've seen is that they are significantly protective, and, for example, reduce your risk of catching SARS by about 70%.
Yes, if a mask doesn't fit properly, some air will come through the sides. Even in that case though, the mask is still filtering some air, and it's protecting you from touching your mouth or nose, and it's protecting you from droplets hitting your mouth or nose from being coughed or sneezed on.