It's not like Antifa is going into people's houses and beating up 50% of the country. If you're marching under torches and hate symbols you don't get much sympathy from me when someone gives you a black eye for it.
Antifa doesn't make you march around with torches and swastikas, you chose to march under the symbols that have been used for centuries for symbolizing intent to murder, lynch, and exterminate "lesser" human beings. Hopefully at that point most people would have a think about "Hans... are we the baddies? We're marching around with torches..."
There are a lot of people who are susceptible to fascist/"alt-right" slogans and thought, even if they're not the ones currently going out in the streets and attacking people. Some of them can be convinced to go a bit farther. "Hey guys, bring tiki torches to the rally tonight!" and so on. It's an active phenomenon with the alt-right to radicalize and recruit additional members - they call it "red-pilling" (after the MRA thing, although many right-wing groups use the term).
It's not dissimilar to the radicalization tactics that Muslim extremists use, which shouldn't be surprising - providing a home and an ideology for the disaffected is a highly effective radicalization tactic.
http://www.dictionary.com/e/slang/red-pill/
https://www.vice.com/en_us/article/nnk3bm/how-to-tell-if-you...
http://nymag.com/selectall/2017/05/the-online-radicalization...
Remember, the actual Nazis were a minority in Germany too, and that took decades to build up.
https://www.cbsnews.com/pictures/white-supremacist-rallies-i...
(it's a sad sign of the times that "Nazis must be and should be stopped by force" is somehow seen as a controversial statement. Sometime, ask a WWII vet if they had any second thoughts about killing Nazis, let alone just giving one a black eye.)
What Antifa does may be illegal, but it's also morally correct. A lot of people have problems with confusing those two concepts.
Keeping torch-wielding supremacists from terrorizing innocents is not morally equivalent with those doing the terrorizing, even if both use violence.