Well this measures how many people are willing to admit they would vote for Trump which is not surprisingly a lower number than people who will vote for him in an unobserved secret vote.

Even if the pollsters are the most neutral nonjudgmental people on earth there will still be a fraction of the electorate that will (irrationally) shy away from telling them the truth.

This effect is certainly real.

Ummm, absolutely not. Pollsters universally stated they either had "adjusted their models" after being embarrassed in 2016, or that the "hidden Trump voter" was a myth (usually combined with "Trump fans won't shut up about how much they support him"). Poll results were repeatedly presented as forecasts of actual election results.

I grew up in a right wing Austrian province which over a decade consistently polled 10 to 15% wrong (in favour of the right wing candidate).

I know those people in person. They are people who will tell even their spouse they vote the left wing candidate, but after the 6th beer they will tell you how they really want the strong guy in charge (note: Hitler is from Austria, voting the rightwing guy can still be a taboo for certain people).

So these people do exist. If it is just them who make up the difference — no idea, but they exist.

I'm not saying such people don't exist -- I'm saying pollsters either claimed to have adjusted the models to account for this phenomenon, or the pollsters claimed they don't exist.

There certainly were some skeptics of the polls, but these skeptics were generally marginalized or dismissed.

The point is, most media outlets who reported poll results did so with the assumption that the poll results would reflect the actual results -- NOT that the polls results needed to be adjusted.

It's clear now that Shy Trump Voters existed. Some opinion pages trafficked in arguments against their existence. I think of them in terms that our Austrian friend described. Some pollsters were better at accounting for them. I avoid "most media outlets" language and try to look at the data.

I don't think they accounted for it. These are people you can't poll

There was one small polling company who claimed they had a methodology:

instead of asking directly who someone would vote for they asked a number of other questions first to warm them up etc.

They predicted way more Trump votes - and were summarily ridiculed by mainstream pollsters I think.

What would you expect of a pollster? "Sorry, we are currently unable to provide satisfyingly accurate polls so we close shop for this election?"

That's a rather defeatist attitude. Perhaps be a bit more creative in how you ask the questions? Find new ways to interact with a more diverse group of people? Take larger samples?

Further, the media could refuse to echo consistently unreliable poll results. Ultimately the media loses when people can't trust what's being reported.

Right, but given how infrequently elections occur, how do you receive the feedback loop to iterate on your models? The reason why statisticians and ML folk like to get huge datasets is that it helps them refine their models. Elections occur every 4 years and often reflect very different political circumstances from previous races. Perhaps if we had continuous polling, then pollsters and models could become more accurate, but as it stands now, most of these models are based on historical data.

Five Thirty Eights' "snake graph", which hasn't changed since before election day, accounted for poll error successfully and predicted exactly which states would go blue and which would go red, along with which several states were uncertain. Some of the uncertains went red, others blue. Florida was the only outlier in precisely how red it went.