Here in the USA, calling someone a Nazi does not at all suggest they are a member of some well-organized noveau-NSDAP. Even the swastika-waving type are understood to be trying to upset and frighten folks.

In fact, given the American love for sarcasm and hyperbole, and lack of an actual historical Nazi party of any note, it seems to me less likely for one to interpret the label literally.