Are you in favor of censoring such speech?
I ask because the only way to get rid of the "consequences" you deride is to introduce large-scale censorship designed to privilege the person who speaks first. Which then turns any debate into nothing more than a race to be first to say something, after which one can censor one's critics under the guise of preventing boycotts, etc.
I am generally in favor of "refusing to associate" since that implies personal choice.
"Holding people accountable" is semantically meaningless since it can be applied to both appropriate and wildly inappropriate actions.
There are also different kinds of boycotts - like, there's a difference between boycotting a book, and boycotting the publishing company of a book. These are more on a spectrum and should be debated on a case-by-case basis. But these days, many social-media-driven boycotts are on the chilling-free-speech side, which is why I spoke generally about them above, even though there are exceptions.
There's a difference between speech, and seeking to punish someone for their speech. Failure to recognize that spirals us to an ever-more authoritarian atmosphere until you find yourself the one being punished.
The comment I initially replied to categorized "speech that demolishes the speech it is responding to". That's still speech. Trying to declare it off-limits, legally or socially, is still an attack on speech.
Similarly, your "punish someone for their speech" is... well, you're condemning people who responded to speech with more speech. Because you didn't like the speech they responded with. There is no way to be a free-speech absolutist and be against boycotts, blacklists, and all the other "authoritarian" stuff you're complaining about, because those things are just as much speech as what they're responding to.
It's illustrative to look at this form of definition when thinking about a boycott, which is an effort to discourage free speech.
"An effort to discourage free speech" is free speech.
"An effort to discourage tolerance" is tolerance. ("You must tolerate my intolerance!")
"An effort to discourage liberty" is liberty.
"An effort to discourage diversity" is diversity. (This is literally the form of argument that the manifesto author engaged in.)
I do not start from a position of absolutism on speech, or tolerance, or a good many other things, and so I happily get to think through the nuances and have a much better chance of avoiding self-contradiction.
This is where Google dude's free-speech defenders get in trouble; many of them want to be, or want to be seen as, free-speech absolutists (if for no other reason than to say they don't agree with Google dude but feel obligated to defend his right to say what he wants). But they also want to condemn people who spoke out against him, called for him to face consequences, called for boycotts and refusals to associate, etc., and cannot do so without being self-contradictory since they themselves need to engage in "speech to discourage free speech" in order to do that.
One of the reasons why even gruesome speech is considered protected is so that people with such opinion can speak out and let everyone know who they are. If someone speaks in favorable terms about sexually abusing children, you will think twice about sending your children to that person's home. However, if such speech is prohibited, you will never know this person's intention and that would put the children living near that person at risk
You personally do not have to debate them, but their speech should still be protected
Sunlight is the best disinfectant