I'm fine with Parler being banned, but for what it's worth:

I think you can be pro free-speech, and still be happy to use the downvote button. The downvote button is an agreed upon and standard way of expressing disagreement on the site, so is in itself a part of free-speech.

That said downvoting a well-reasoned, but contrary-opinioned comment is clearly against the intention of the button, which weakens my argument a bit.