If saying something can essentially wreck your life and turn you into a pariah because you can't even get a job, what does it matter if it's a government or not doing it? When the church punished heretics, did it also not count as a violation of freedom of expression because it was not the government doing it?
Conversely, it's impossible to have a "marketplace of ideas" if all forms of criticism, regardless of their source or intent, are considered a violation of someone's freedom of speech, and this presents the paradox of only considering active speech to be defensible, but not reactive.
A marketplace of ideas implies that some ideas will be considered not worth buying.
Criticism is fine, trying to shut opponent's mouth is not, punishment is not.
Now define "criticism" in a way that can't be arbitrarily equated to "punishment" or "trying to shut your opponent's mouth."
Demanding the person to be fired is not fine, ad hominem attacks are not fine, sharing private information about the person is not fine. You get the idea.
To claim otherwise would seem to place business and company culture beyond the reach of the public interest and the scope of society, which is a position not even free governments enjoy.
Perhaps, you are confusing your own interest with the public's one?
The "right to be forgotten" that is being recognized in some countries is also an example of a government measure that can help.
By the way, you seem very confident that nothing should be done because these things only affect people with "awful" or "shitty" beliefs. But what is socially considered awful can vary a lot. There are societies that would place being gay, abortion or being an atheist under that umbrella; conversely, other societies in the past placed being religious under that umbrella. Even if you don't like the people being affected by this now, allowing freedom of speech to be eroded in this way can backfire in a big way in the long term.