Re:One pollster’s explanation for why the polls got it wrong
by colinmhayes on Wednesday November 11, 2020 @02:41PM
(#25062068)
The badly worded question issue is clearly major, but it's received so much publicity, and with meta pollers like 538 existing it seems like it shouldn't be a huge problem when there's such a wide selection of big money polls available. All those ideas seem decent, but as you said they're not really solving the problem, just inching you toward a less biased sample. I'm sure people that already declined to take the poll are real happy to be bothered again about it. As far as I can tell the only way to actually make polling work is to force people to respond, and obviously that won't happen. And even if it did people would give troll answers(at school we had captive polls and the last question was always "how seriously did you take this poll?" of course the least serious people always marked themselves as most serious). Even paying people to respond biases the sample because now you're more likely to get people who want money(not necessarily totally correlated with income). Of course you can infer that your sample is biased by one side being more enthusiastic, but without a way to enumerate how biased you're nowhere.
My job is basically data analyst/programmer, but I have a decent background in research of this sort (not politics, surveys and qualitative research) and occasionally have to conduct surveys. It often falls to me, or I insist on it, because practically no one knows how to write quality survey questions that aren't leading questions that bias the response.
For example, if you wanted an opinion on a new food product, someone that doesn't know how to ask a neutral question (or is purposely rigging the survey) could ask "How much do you like these french fries? 1) Fantastic 2) Very Good 3) Good 4) Okay 5) Awful. Those answers are front stacked with positive answers, only one negative answer, and no neutral answer. If someone doesn't believe they're awful, they have no choice but to answer "okay".
In any case, one common method I use to measure the presence of responder selection bias is a followup communication to non-responders 1-2 weeks later to nudge them to take the survey. The hypothesis is that people who did not response initially, but do respond when prodded again, will represent a middle point between initial responders & complete non-responders. If that middle point is not statistically significant in their responses that the original responders, I can infer that non-responders are less likely to be a group that would answer differently if had bothered to answer.
No, this is absolutely not a perfect method. More robust methods are needed to further reduce uncertainty: Ethnographic & other qualitative studies can help. Talk to responders to find out what they responded and consider the inverse. e.g., if they say "this is a really important election we have to get Trump out and I want to spread the word so it was important to share my opinion" that should make you question whether Trump supporters would feel the same way about responding.
To really get to non-responders, you need more time & more resources. Find them and pay them to talk to you. It's not uncommon to reward research participants who certainly wouldn't participate otherwise.
Again, none of these are perfect, they just inch you close to greater understanding a little bit at a time.