Lots of people — especially Jews! — have spent enough time on this to figure out indications that someone might be a Nazi. These guys looked like Nazis, and guess what: they turned out to be Nazis.

It’s not “assuming guilt by association”, it’s pattern recognition.