Ideologically, they're all extreme.

Practically, one of those three groups were widely successful among many millions of people. So you'll find a great sensitivity to that one.

The people that the left calls neo nazis would likely reject that label.

Just because you call a group you don't like a Nazi, doesn't make it true.

Doesn't make it false either. There is significant overlap between established neo-Nazi movements and the "alt-right", both in membership and rhetoric.

Where else are they going to go? There is a "significant overlap" between social justice and anti-white, anti-male, anti-cis propaganda. There's no convenient label to stick on that one just yet, but if you want to be consistent you're going to have to paint all social justice people with that brush.