Universal free speech as it stands, allows racism and with that, you are not wrong. Free speech within some moral boundaries, though free debate, that is the aspect of free speech to provide balance that is equally important.
When you do start censoring them, we will all start losing faith in our values as we don’t apply it across the board.
If you look at other non free countries, the average person doesn’t really believe that you have the right to express your views. Take the Muhammad cartoons for example, a vast majority of Muslims in non-western countries think that’s a no go from a rights perspective ( as in, you shouldn’t even have the right to draw it). That kind of thinking extends and extends to all kinds undermining of individual rights. If you look at China, the vastness of inaction to their Internet being censored is another key example of this. It takes a lot to get a society to value it, and we worked pretty hard to get those values ingrained in our culture.
There might be some absolutely abhorrent people in our country, but even they cannot deny that they do have the right to express themselves here. We lose that baseline integrity of our values when we start withering it away, until we ourselves can’t even recognize ourselves anymore, where suddenly we have all these caveats to a principle that defines our society.
I genuinely don't think universal free speech is the answer, because people aren't confronted with the counterargument until they seek it out. If you could guarantee that argument, it would be much safer. But I don't think it's possible with today's media; people exist in a bubble of both their own choosing and algorithmic choosing, and aren't challenged.