The first amendment to the US Constitution only applies to the US Government. The government cannot prevent you from spewing your bile. It in no way applies to interactions between individual citiizens. You a dreadfully mistaken if you think it's a free pass because "muh free speech".

Additionally, hate speech is also violence, and you cannot hide behind free speech to say "but look, you're doing physical violence to me". So, yeah, beat up nazis.