There are ways of correcting for non-responders though. For example, there is the Monitoring The Future study that asks high school kids about their history of drug use. Almost everyone who is in school that day answers the questions, but that still leaves everyone who was absent that day. So what they do is they ask the kids the number of days of school they have missed so far this year, and then they correlate the number of missed days with drug usage. Using this information they can correct for every kid who was absent, because they know how many absentees there were and can figure out the distribution of how many of those kids have skipped one day of school so far, two days of school so far, etc. And then they can assign drug usage rates accordingly.
This is very different, because while there is some data missing, it is missing at random (albeit weighted), and can thus be corrected for.
If it was the case that kids avoided coming into school that day, because they knew the survey would be held, it would be a very different problem.
There is an entire literature on the validity of self-reported drug use that they take into account. For example, there is a phenomenon called recanting, where someone will admit to using a drug one year and then deny it in future years. They correct for this by ignoring when someone recants if they have already admitted to using the drug on at least two previous surveys. (Which would be administered at least two years apart.)
What they found was that the people most likely to recant were the ones who got jobs where they would now be fired for admitting drug use. For example, cops recanted drug use after previously admitting to it at twice the rate of the general public. So that's how they made the decision to discount recanting for people who had already admitted to drug use twice or more times.
As for people who try to lie and claim they are taking drugs that they haven't been, this is corrected for by asking about the same drug multiple times in different ways. I think they may also salt the questions by asking about drugs that don't exist.
You can obviously read all about the methodology of Monitoring the Future by downloading it, and here is an enormous PDF with tons of studies on the validity of self-reported drug use in general:
http://archives.drugabuse.gov/pdf/monographs/monograph167/do...
All in all the methods they are using seem fairly accurate. It would definitely be a little higher if they corrected for more stuff, so for example when they say that 82% of Americans have used marijuana, it's actually probably closer to 85 or 86% when you correct for additional things that they haven't.
The research seems to show that the self-reported responses are pretty accurate. For example, the number of people who reported using LSD basically went to zero right after the Pickard bust in 2003, when pretty much the entire US production got shut down. This means that if there are people lying about taking drugs they haven't been taking, it can't really be much than a couple tenths of a percent.
In this case, there's a strong argument to be made that the samples could skew young (if for no other reason than because younger people frequent sites like HN more than older people). It's also quite possible that the demographics for HN and for slashdot are different in a way that would make the HN audience skew younger. But the voluntary response process itself? I think it's a stretch to claim that it'll cause a measurable bias in age distributions, across two different surveys, spaced a decade apart. You'd have to argue that younger people in 2011 are more likely to answer a poll than younger people in 1998 were to answer a poll. Possible? Sure. The most likely source of error here? Nah.
In other words, if you're going to pick (and you should!), pick on the BIG, OBVIOUS differences between the data sets...not the little ones.
I personally can't see significant problems of this over any survey technique.
There are problems with questionnaires and self-responses, sure, but to say they are worthless is to exclude an entire category of data based on a selective opinion. Horses for courses, if you like.
Many statistical methods are available, some of which are more appropriate than others in particular situations. There is little wrong with convenience sampling if what is of primary importance is convenience, e.g.
(That being said, I'd be careful with taking the results of this poll and applying it to any technology site and therefore the interwebs etc. Inferring from this data that we don't have an ageing population problem would be dangerous).
-----Original Message----- From: Paul Velleman [[email protected]] Sent: Wednesday, January 14, 1998 5:10 PM To: [email protected]; Kim Robinson Cc: [email protected] Subject: Re: qualtiative study
Sorry Kim, but it just aint so. Voluntary response data are worthless. One excellent example is the books by Shere Hite. She collected many responses from biased lists with voluntary response and drew conclusions that are roundly contradicted by all responsible studies. She claimed to be doing only qualitative work, but what she got was just plain garbage. Another famous example is the Literary Digest "poll". All you learn from voluntary response is what is said by those who choose to respond. Unless the respondents are a substantially large fraction of the population, they are very likely to be a biased -- possibly a very biased -- subset. Anecdotes tell you nothing at all about the state of the world. They can't be "used only as a description" because they describe nothing but themselves.
http://mathforum.org/kb/thread.jspa?threadID=194473&tsta...
For more on the distinction between statistics and mathematics, see
http://statland.org/MAAFIXED.PDF
and
http://escholarship.org/uc/item/6hb3k0nz
I think Professor Velleman promotes "Voluntary response data are worthless" as a slogan for the same reason an earlier generation of statisticians taught their students the slogan "correlation does not imply causation." That's because common human cognitive errors run strongly in one direction on each issue, so the slogan has take the cognitive error head-on. Of course, a distinct pattern in voluntary responses tells us SOMETHING (maybe about what kind of people come forward to respond), just as a correlation tells us SOMETHING (maybe about a lurking variable correlated with both things we observe), but it doesn't tell us enough to warrant a firm conclusion about facts of the world. The Literary Digest poll
http://historymatters.gmu.edu/d/5168/
http://www.math.uah.edu/stat/data/LiteraryDigest.pdf
is a spectacular historical example of a voluntary response poll that didn't give a correct picture of reality at all.