Depends on what they're swinging, if they are swinging a lead pipe or brick it would absolutely be an act of self defense. Just a fist meh I don't see it happening unless there are multiple assailants attacking you