I know those people in person. They are people who will tell even their spouse they vote the left wing candidate, but after the 6th beer they will tell you how they really want the strong guy in charge (note: Hitler is from Austria, voting the rightwing guy can still be a taboo for certain people).
So these people do exist. If it is just them who make up the difference — no idea, but they exist.
There certainly were some skeptics of the polls, but these skeptics were generally marginalized or dismissed.
The point is, most media outlets who reported poll results did so with the assumption that the poll results would reflect the actual results -- NOT that the polls results needed to be adjusted.
instead of asking directly who someone would vote for they asked a number of other questions first to warm them up etc.
They predicted way more Trump votes - and were summarily ridiculed by mainstream pollsters I think.
Further, the media could refuse to echo consistently unreliable poll results. Ultimately the media loses when people can't trust what's being reported.