It is protected. It's just that hate speech is an explicit exception from that rule (which is the whole point of that designation). Most of hate speech is political in some way.
So it all boils down to who gets to define what is "hate speech", exactly.
> I still don’t agree the trade off for Nazis is worth it.
What trade-off? Neo-Nazis flourish in the very same countries that have extensive censorship of their ideologies and speech. Look at Germany with their "streitbare Demokratie" for an example - they don't just censor speech, they outright ban parties solely on the basis of their platforms. And for those efforts, they get yearly neo-Nazi marches that are several times bigger than the one-off we've seen in Charlottesville, and AfD is now the second most popular party.
On the other hand, there's this trade-off:
https://theintercept.com/2017/08/29/in-europe-hate-speech-la...
To give just one example, in France, it is currently illegal to wear a t-shirt that says "boycott Israel".
https://forward.com/news/breaking-news/323207/france-court-u...
So yes, USA is an improved society in that regard because of its strong freedom of speech protections. Hopefully we can keep them.