Swastikas are a little different though. It's been decoratively used for apolitical reasons for thousands of years on all sorts of things, but was only really flown as a flag starting with the NAZIs in the 1920s. There's a lot of architecture, even in Europe involving pre-NAZI swastikas. And before and even shortly after the NAZI party adopted it, it was used a a good luck symbol in the west (http://en.wikipedia.org/wiki/Western_use_of_the_swastika_in_...).
Even more extreme: Swastikas in the decorative masonry in a government building? (http://en.wikipedia.org/wiki/File:Greca_a_svastica_-_palazzo...) Don't expect much controversy. Swastikas on a flag in the same building? Expect an issue. Something being in a flag or not is an incredibly important context that must be considered.