Freedom of speech is not violated by ostracism, because freedom of speech (as a right) is the right to say what one wants, to whom one wants, at a time of one's choosing and the implied listener's freedom - to listen to whom one wants (or not) at a time of one's choosing (and hence to what one wants).

If someone is ostracised that is an example of the listener exercising their free speech rights, not a violation of them.

Some use this as an example of why companies like Twitter and Facebook may remove someone from their platform, but I would argue firstly that they are a monopoly exercising monopoly power, hence violating one's rights to free speech, and that they are not platforms any more because of their interventionism in the speech of others (and currently have too much protection under law to do this). The monopoly power is the most important thing to challenge, in my view.