No decisions on work-specific matters should be taken on non-mandatory social events were some team members can't/don't want to attend. If you do it, fill the missing people in the next day, the same way someone was missing from an important meeting for whatever work-related reason.