That is not my experience at all. Sure, some people will feel uncomfortable, but it shows right away, and I stop. Some people are more open and return the joke, keep the conversation going, etc

That's always nice, for everyone involved (or so I like to think).