That symbol is associated with mass murder of millions and a world war.
Previous to that, it was not a common, everyday symbol used in western culture.
It's not a dumb little joke from 2017 from 4chan.
The Nazis had a hand gesture: the raised arm Roman salute. You can use that salute today without being dubbed a supremacist or Nazi. Just maybe don't shout heil anything while doing it.
Also I am not saying every use is racist or anything that extreme, but it is clear that in some instances there is a connection between this symbol and white supremacy.