I think being sober around drunk people is like going to a naked beach with your clothes on. It just doesn't feel right, and if you don't want to drug yourself up then you shouldn't want to be around people who are.

It won't feel right if you're trying to fit in the same way, because you fundamentally are lacking the necessary attributes to do so (drunkeness, lack of clothes), but that doesn't matter if you don't feel a need to fit in. If you have any interest in people-watching, or perhaps one might call such interests anthropologist tendencies, things are fine. Observe, interact and join conversation or events as you please.

Besides, most people are multi-dimensional, even if you don't fit in on this thing, you might fit in on another axis. Why are you even with the group to begin with? Focusing on that will be more important for social bonding than focusing on joining or not in alcoholic drinks.

Or just abstain from groups, but that comes with its own costs

Or going to a normal beach with a hijab? If your friends can't respect you deciding not to drink, it's much like being uncomfortable with you going to the beach with them with a hijab.