> Rather, I was saying criticizing the idea of restricting speech is not inconsistent with free speech ideals

Restricting in what way though? Formal restrictions: laws, assuredly this is inconsistent. But if the "restriction" not a formal restriction, but instead a reaction to criticism, then this is a meta-criticism of criticism, and that is counter to those ideals, because criticism itself is speech.