If people harass you or treat you differently for not [drinking/eating/smoking/whatever personal choice] then you know who the assholes of the group are. Call them out.