Without digging too deeply into semiotic philosophy, symbols are also defined by those who use them. 99% of people brandishing Nazi flags have horrific views, and use the flag to show their allegiance to those views, so we prefer people not to brandish them. Same thing for confederate flags