It's stronger than simply "trolls". "Alt-right" is just the latest PR rebranding used by neo-nazis/white-supremacists. Obviously, generalizations about any group of people is problematic; trolls and other not-explicitly-neo-nazi people use the label "alt-right".. Those people might want to consider that if you hang out with neo-nazis and repeat their memes/trolls, people might start to assume you are a neo-nazi. If someone likes to spend time in hang out with ${sport} fans, uses slang associated with ${sport_team}, most people are going to assume they are fan of that sport even if they claim* otherwise.

I suspect a lot of people might think I'm misinformed because someone in the "alt-right" said they were not nazis, and would never support racial violence. Remember that neo-nazis lie. Is your opinion abased on their public PR-focused interviews? If you have any doubt whatsoever that these are neo-nazis dreaming of bringing back the holocaust, I suggest watching this[1] video at 41:12, which is footage from a popular "alt-right" discord stream inside the Charllottesville rally. In their own words:

    Speaker: Did Hitler do anything wrong?
    Audience: [loud "no"]
    Speaker: One more, I want everybody to repeat after me..,
             ,,,this is the first precept of the true alt-right:
             Gas the [slur]s, race war now!
    Audience: [even louder cheering]