That's the problem with FPTP systems. In true multi-party proportional systems, people can align with the candidate on much more than few extremely polarizing points. Also gets rid of most effect of gerrymandering.

On the other hand, later coalition building often results in significant compromises too.