I hear where you're coming from. I used to feel this way, and I sometimes catch myself falling back into this thought pattern.

But now I try to be as open as is practically reasonable about my interests and hobbies. You'd be surprised how receptive some people can be. And those who react negatively are filtering themselves out of further interaction.