People live so much of their life in virtual worlds of media & thought that they confuse them with reality. So people will decide that they dislike Mormons (because everyone knows theism is dumb and antiscientific) or Muslims (because they're all terrorists), despite never having truly met any, having zero knowledge of their history, etc. The negative reactions you speak of aren't about Mormons or the church, they're about figments of sequestered imaginations, unconstrained by reality.

It's a shame. Real life is so much richer. In my real life I've often come across Mormon missionaries (in the past knocking at my door - not sure if they still do this?), and despite radical differences in view, have mostly had warm & engaging interactions.