Same, I only pull it off when i feel it scratching a palm rest.

Smaller watches, with nylon, silicone, or leather straps help. Big metal bracelet watches always feel in the way.