Yes, Mill's general principles for liberty exclude the right to harm others. But he's very very skeptical of considering speech a harm. He's not a counterexample to the literal sense of what you wrote upthread about "saying whatever we'd like without fearing repercussions": e.g. you should be legally liable for libel, and socially liable to individual opprobrium. But he was clearly against the kind of repercussions for the kind of speech we're talking about here.
Okay. So if we're preventing harm, then, what is the possible harm here? Do you think intolerant beliefs threaten society? I'm of the opinion that beliefs mean nothing until acted upon, that only intolerant actions threaten society.
Or to rephrase it as one of the core tenets of Silicon Valley, "Having a great idea is not enough. It is the manner in which ideas are executed that counts."[1]
1: https://news.ycombinator.com/item?id=2318849, linking to https://archive.is/U6C4l#selection-2043.104-2043.196
The part you haven't come to yet is the Harm Principle.