I don't think it's the polls on their own that are the problem. I think it's the media-driven furore around them that feeds into the public's pre-existing confirmation bias ("see, the polls prove he's an idiot with no chance at the presidency!").

It seems like most people on both ends of the political spectrum create a media-fed bubble of bias, and when the outcomes disagree with that bias (as in the case of Trump) it can create pretty damaging cognitive dissonance.