Just calling yourself a Nazi or doing the chants or whatever carries with it an implicit provocation and threat of violence toward minority groups.

Responding with actual violence in turn is not the right approach, but when we see large armed mobs forming and declaring themselves pro-genocide, it's absurd to call the people protesting them the 'real problem'.