Totally agree with your last few sentences.

I don't think anyone is proposing making it illegal to boycott, it's just counter to a culture of free speech to try to punish people because you disagree with their opinions. That culture isn't legally protected, except the part where the government can't infringe on it.