Yeah, same, I would also hang out with smokers during their smoke break at a job, despite the fact that I have never really smoked [1]. At that job, most of the higher-ups smoked and I figured it might be useful to be part of the "under the table" discussions.
[1] Outside of half a cigarette when I turned 18 which I thought was pretty gross.
From what I’ve seen (in the US), people will frame in like it will hurt, because drinkers what to drink with other people, but competence wins out at the end of the day.
Being around when decisions get made outside of the office can have an impact though, and it sounds like you’re still there for that. I used to go out with the smokers on their smoke break for the same reason. I didn’t smoke, but going out with them let me be part of the conversation that was sometimes social, but often turned to work and strategy.