Right, but even with a cotton mask the worry is hand-to-mask, not the other way around.

When you're out in the world, the potential concentration of viruses will generally be much greater on your hands than on the outside of your mask.