Are you saying that it's the fault of the groups not directly present, we can blame the moderates who aren't there?

You'll either eject the cranks or become them. The (possibly apocryphal) tale is "As we say in Germany, if there’s a Nazi at the table and 10 other people sitting there talking to him, you got a table with 11 Nazis."

This dynamic has happened many many times on the left, with the far left cranks.