The complicated part is that freedom of speech is both a legal right in the US and an ethical principle globally, and your legal rights are less broad than the ethical principle. Something that is legal can still be in violation of the ethical principle.

You say "[we] don't get to choose [people's reactions]", but this is not at all relevant. We're not asking whether their actions are legal or whether we can legally dictate what they think, we're asking whether they're acting ethically.