When there's obviously no neonazi intentions there's nothing wrong with swastikas. Red black and white are pretty basic colours, you can't suggest that either.

There's something wrong with the people who'd make such conclusion, and not self-censoring will help get rid of connotations.