I think a large part of the issue is that people that lean slightly one way are getting lumped in with the extremes of that direction where they don't see themselves like that.

Someone that questions immigration laws isn't necessarily a white supremacist for example, but that narrative isn't as interesting (doesn't sell as much advertising) as the white supremacist one.