People touch their faces all the time, whether they're wearing masks or not. This theoretical harm exists equally in both cases; in fact, touching the mask may remind you you're not supposed to be touching your face, and trigger a hand washing or hand sanitizer use.