Viruses may still be infective if captured by the mask, but they're much less likely to infect you. If they're captured by an intermediate layer of the mask, they pose no threat. Even on the surface of the mask, the chance of you being infected by a virus particle that you touch on the surface of a mask when you take it off are overwhelmingly less likely to infect you than virus particles that you breathe in. Of course, when removing a mask, you should sanitize your hands afterwards, which nearly eliminates the risk from virus particles you may have picked up by touching your mask.
You should not pull your mask below your nose. If you do however, the chance of a virus particle coming off the mask and then going into your nose is dramatically less than if you had been breathing in these particles directly.
Viruses are still infective after being captured in the mask. Touched the outside of your mask? Go wash your hands. Mess with the mask? Go wash your hands. Does the mask make you touch your face more? That's a huge risk. Does it unseal a little when you talk or move your mouth? Air will go in that way, and virus will come with it.
Moreover, if you are like most people I see on the street, and you push the mask down below your nose or mouth to talk, you are literally putting a super-concentrated source of virus right in front of your face. Most people, most of the time aren't exposed to enough virus to get sick. Breathe through a mask all day long, and then yank it down below your nose, and you're creating a problem. You have to be careful about how you put it on and take it off, especially if you're re-using a mask.
None of this is rocket science, but almost nobody gets it right without training. Even then, it's hard. Half of the reason medical staff wear face shields is to prevent them from screwing with their masks or touching their face. It isn't trivial.