Feminism is an idea. Feminists, or the people who adhere to that idea, are a group. If every single adherent of an idea you encounter is an abhorrent person, most people will stop and consider if they want to associate with those people. Typically the answer is "No".

Most people will also wonder if something about the idea turns people into abhorrent people or if something about the idea is only attractive to abhorrent people. In either case, it is not something to which our notional person is likely to be attracted.

In short - the people who adhere to an idea are seen as commentary on the idea.