You're not a weirdo but yes, lots of people struggle saying 'no' to a drink.

Your caffeine example is a good one. To some folks on /r/stopdrinking, alcohol is more akin to how you feel about caffeine.