Note that the mainstream Fediverse's federation isn't just refusal to "associate with Nazis". If you maintain a server and are not a Nazi yourself, but you federate with a third server where a handful of people said something offensive (and this got blown up in the media even though it was just a handful of nobodies), the mainstream Fediverse will refuse to federate with you.

It gets to the point where the Fediverse ends up enforcing a pretty narrowly defined orthodoxy not reflective of even many traditional anti-Nazis. For example, the Fediverse has a conspicuous numbers of strident trans activists in influential positions, and e.g. posting the sort of views that J.K. Rowling held (shared by many intellectuals who consider themselves on the Left) may to outrage and refusal to federate.