You are confusing laws (like 1st amendment of the US Constitution) with freedom of speech. Laws exist because freedom of speech is important, not the other way around. The fact that laws only restrict the government from suppressing freedom of speech doesn't mean it is not important in other contexts, just because there's no law enforcing it.
> nothing can compel me to listen
Banning someone from the internet (or any part of Internet that is serviced by a major corporation, which is almost all of it) is not the same as "I do not want to listen", it's "I don't like you so nobody gets to listen to you, at least not without extraordinary effort which 99.999% of people are not capable of". If you just don't want to listen to somebody on youtube, it's enough to not go to that channel. Removing the whole channel goes way beyond that and the only purpose of that to silence, not to "not listen" but to cause others not to listen.
freedom of speech does not keep someone from thinking you (not you specifically, but the general you) are an idiot or a kook, nor does it protect you from the consequences of your words by those other individuals (obviously other laws protect you from intentional physical harm and the like). nothing can compel me to listen to you or make a company broadcast your conspiracy theories (again, the general you).