No, it is not. You have the freedom to rebuttal, but not the one to insist of speakers to be silence or removed. Look up the definition of freedom of speech on wikipedia, it is in the first sentence.

This is not absolutist at all, this is a very basic rule for civilized discourse. You are correct that people occasionally do not adhere to these principles.