It's pretty easy to not touch your face when your hands are physically on the handlebars. As long as you are able to wash your hands immediately after getting off the bike, I think you are reasonably safe.