The polls weren't even that far off, on a national level they were basically spot on, the organizations reporting on them just threw out the data that made it look like Trump had a pretty good chance. 538 did a slightly better job but the editorial bias in the presentation of the data was distinct across most of the media.

I don't think it's the polls on their own that are the problem. I think it's the media-driven furore around them that feeds into the public's pre-existing confirmation bias ("see, the polls prove he's an idiot with no chance at the presidency!").

It seems like most people on both ends of the political spectrum create a media-fed bubble of bias, and when the outcomes disagree with that bias (as in the case of Trump) it can create pretty damaging cognitive dissonance.