The more likely time is when stopping for a minute to look at the map or change playlists. It is obnoxious to always be taking the gloves off for it.

I don't like to talk on the phone when I'm not riding; why would I ever choose to do it during my decompression time? :)