That's sort of the question isn't it. Is it that there are so few of these neutral grounds that they collect the people unwelcome anywhere else, or that those ideas out-compete everything else unless there's a finger on the scale?

Personally I think it's a hybrid. Extremism and outrage are memetically advantaged in all circumstances but the specific atmosphere is what chooses the flavor prevalent in a particular community. Voat was formed by far-right elements and so that became the culture.