Part of the problem is that those organizations allow fascists to enter them. (Like many left wing movements allow authoritarian communists -- tankies -- to enter them.)

If a part of your platform is "No immigration" it becomes very difficult to distinguish that from "No immigration (because I don't like brown people)".