Extremists by definition are a small fringe faction. This situation is similar to thousands of other gestures and symbols, even though the vast majority do not recognize or care about what some extremists use. Literally everything has a extremist use somewhere.

Also I'm from India and as I just explained, the Indian swastika is a different design and direction. The Nazi swastika is not used by anyone else, they made it for themselves which is why it's only associated with them.

If you won't listen to the majority group that uses the term telling you it's positive for them, and instead apply the extremist interpretation, then what exactly do you need to hear as a counter position? Seems like it's not open to any other understanding.