Women and minorities have no obligation to enrich someone working against their interests. The only "silencing" going on is the universal desire to not work for a jerk.