That's because you've never been in the reverse situation. Think about not drinking at weddings, your birthday, Christmas, family gatherings in general, milestone celebration at work, and lots of other occasions where people typically drink.

I can assure you someone will ask if you're okay, urge you to drink "at least one", or "just a glass of Champaign, it's a special one".