>I don't know how you make something capable of user interaction AND immune to keys, pocket fuzz and soapy water. It's a tall order.

One consumer device that has consistently delivered that is the waterproof, rugged feature phone. In fact, taking an old Samsung B2100 and writing a Java app to serve as a Bluetooth HSM probably leads to a cheap, serviceable solution.