And potentially loosing your job is quite a big deal for most people. That's what happened to James Damore for bringing up the wrong subject.
Not saying you think muslims are dangerous is at the same courtesy level as not telling your coworker his hair looks like shit every day. Nobody is going to (immediately) fire you for it, but people will definitely start avoiding you if you do it to an obnoxious degree.
If Christians were targeted by violence on American streets after 9/11 (as they were--along with, you know, completely unrelated Sikhs), "the left" would say that's wrong, too. And the reason that the status quo with regards to anti-Islam sentiment is what it is is because Americans in a whole lot of places genuinely haven't progressed much beyond that mindset.
There's a difference of kind here.