Touching others (in whatever form) is different than "holding hands" in the way some cultures do. The difference is that there are hundrends of ways to touch someone, and thousand of circumstances where you do so. So any particular form is just one of many.

On the other hand, you carry your face and its' facial expression constantly, and your expression is the most basic form of interaction you have with another person socially, with talking as distant second. Furthermore, there are few expressions, are (almost?) all are universal, like smiling and crying.

Heck, you could live in the same room with a person for a month, with all kinds of normal interaction (eating, talking, etc) and never one touch him/her, and much less in any socially meaningful way (like holding hands or hugging). Not at all like avoiding having your face seen.