Touching it at all is likely a deal-breaker. People would likely prefer using a 'remote' on their phone to issue commands, rather than using an indirect pointing/clicking device on their temple.

Mobile is primarily different from Desktop in that the little annoyances are too much for the form factor and use cases. And indirect pointing and clicking just doesn't cut it in mobile.

Wearable will only be more extreme this regard.