The issue is not that people are criticizing people who have certain ideas. That is fine. That is free speech. The issue is that people are taking actions that actively harm those people. Whether by excluding them from events, preventing them from speaking, firing them from their jobs, etc.

If you fight words with words, that is good. If you fight words with actions against the people who said those words, then you are against free speech.

Note: the principle of free speech is much broader than just the Bill of Rights.