Of course it is abusable. The state has done all sorts of tremendously evil things when wielding its power. "Free speech" has never stopped the crushing of legitimate protest. This is not a surprise to leftist activists, who have been consistently violently attacked by the state for more than a century. I find these arguments so disingenuous when people throw them out while also doing little to nothing to curb existing state abuse. I've never seen right wing "free speech absolutists" at trans-rights protests or fighting against state interference in left-coded academic work.

General restrictions on state power don't work. Capitalism crushes people. When that state power turns on the majority, the majority complains that their voices are silenced and then they go right back to ignoring the continued problems of the minority.

Perhaps the solution is to approach each situation on a case by case process. Protest and agitate when the state harms people and celebrate when the state protects people. Make harm reduction the goal.

In software people often point to how focusing on metrics can make you fail to achieve your true goal. Similarly, we should focus on the thing that free speech is intending to achieve rather than free speech itself.