I've found a useful model is to think of your behavior as a selection-bias filter on the people you hang out with. I don't think it's reasonable to expect everybody to like what you're doing. People don't even agree with each other, so if nobody disapproves of what you're doing or how you conduct your life, chances are you're not doing anything important or actually having a life. A group where everybody agrees on everything is a cult, not a culture.

But if nobody likes you and wants to hang out with you, you have a problem. And if the people who like what you're doing are people that you yourself don't really like, you're probably not being true to yourself. You want to be in a situation where there's a core group of people you like and respect who also like and respect you. If you've got that, who cares what other people think?