Yes, you should be allowed to call him that. No, you should not be allowed to punch him in the face, and should be arrested and jailed for doing so.

Violence is justified to protect yourself or others from violence. Not from speech, from violence. There are some small grey areas, but this isn't rocket science.