99% of people don't know what fascism (Italian fascism, I suppose, not national socialism, Falange Española or BUF) actually is and only mean "some kind of vaguely traditionalist (but not monarchist) authoritarian" by it. It's actually a pretty good red flag for people who shouldn't be trusted to discuss politics.

There are actual fascists (and not as few as I would like) and they need to be called out, but using the term inaccurately and provocatively on a broad group makes it easier to oppose the usage outright. Optics are important to politics, like it or not.