News for hackers. Stuff that matters.
Re:Poll: How old are you? by timr on Thursday February 03, 2011 @03:55PM (#2176761)
Interesting comparison from 1998:

http://slashdot.org/poll/421/My-Age-is

In 1998, 22% of respondents were ages 16-20, 53% were in their 20s, 15% in their 30s and 5% over 40.

As of this writing (and assuming that I did my arithmetic correctly), ~7% of respondents on this poll are ages 16-20, ~60% are in their 20s, ~25% in their 30s, and ~7% are over 40.

Obviously there are a lot of caveats here, but the slight increase in the overall age of techies between these two polls is nothing like you'd expect if everyone who started in their 20s in 1998 stuck around until today.

*edit: fixed arithmetic error. D'oh!

Re:Poll: How old are you? by tokenadult on Thursday February 03, 2011 @05:14PM (#2177163)
As I commented previously when we had a poll on the ages of HNers, the data can't be relied on to make such an inference. That's because the date are not from a random sample of the relevant population. One professor of statistics, who is a co-author of a highly regarded AP statistics textbook, has tried to popularize the phrase that "voluntary response data are worthless" to go along with the phrase "correlation does not imply causation." Other statistics teachers are gradually picking up this phrase.

-----Original Message----- From: Paul Velleman [[email protected]] Sent: Wednesday, January 14, 1998 5:10 PM To: [email protected]; Kim Robinson Cc: [email protected] Subject: Re: qualtiative study

Sorry Kim, but it just aint so. Voluntary response data are worthless. One excellent example is the books by Shere Hite. She collected many responses from biased lists with voluntary response and drew conclusions that are roundly contradicted by all responsible studies. She claimed to be doing only qualitative work, but what she got was just plain garbage. Another famous example is the Literary Digest "poll". All you learn from voluntary response is what is said by those who choose to respond. Unless the respondents are a substantially large fraction of the population, they are very likely to be a biased -- possibly a very biased -- subset. Anecdotes tell you nothing at all about the state of the world. They can't be "used only as a description" because they describe nothing but themselves.

http://mathforum.org/kb/thread.jspa?threadID=194473&tsta...

For more on the distinction between statistics and mathematics, see

http://statland.org/MAAFIXED.PDF

and

http://escholarship.org/uc/item/6hb3k0nz

I think Professor Velleman promotes "Voluntary response data are worthless" as a slogan for the same reason an earlier generation of statisticians taught their students the slogan "correlation does not imply causation." That's because common human cognitive errors run strongly in one direction on each issue, so the slogan has take the cognitive error head-on. Of course, a distinct pattern in voluntary responses tells us SOMETHING (maybe about what kind of people come forward to respond), just as a correlation tells us SOMETHING (maybe about a lurking variable correlated with both things we observe), but it doesn't tell us enough to warrant a firm conclusion about facts of the world. The Literary Digest poll

http://historymatters.gmu.edu/d/5168/

http://www.math.uah.edu/stat/data/LiteraryDigest.pdf

is a spectacular historical example of a voluntary response poll that didn't give a correct picture of reality at all.

Re:Poll: How old are you? by Alex3917 on Thursday February 03, 2011 @05:30PM (#2177227)
"voluntary response data are worthless"

There are ways of correcting for non-responders though. For example, there is the Monitoring The Future study that asks high school kids about their history of drug use. Almost everyone who is in school that day answers the questions, but that still leaves everyone who was absent that day. So what they do is they ask the kids the number of days of school they have missed so far this year, and then they correlate the number of missed days with drug usage. Using this information they can correct for every kid who was absent, because they know how many absentees there were and can figure out the distribution of how many of those kids have skipped one day of school so far, two days of school so far, etc. And then they can assign drug usage rates accordingly.

Re:Poll: How old are you? by feral on Thursday February 03, 2011 @08:23PM (#2177890)
Their ability to correct for the missing kids, is presumably because they can assume that the kids missing are missing at random (proportional to their attendance rate.)

This is very different, because while there is some data missing, it is missing at random (albeit weighted), and can thus be corrected for.

If it was the case that kids avoided coming into school that day, because they knew the survey would be held, it would be a very different problem.

Re:Poll: How old are you? by obfuscate on Thursday February 03, 2011 @07:23PM (#2177696)
Well, they can assign reported history of drug use correctly. Do they do anything similar to correct for lying, or just label their data as "self-reports of drug use"?
Re:Poll: How old are you? by Alex3917 on Thursday February 03, 2011 @08:32PM (#2177919)
"Do they do anything similar to correct for lying"

There is an entire literature on the validity of self-reported drug use that they take into account. For example, there is a phenomenon called recanting, where someone will admit to using a drug one year and then deny it in future years. They correct for this by ignoring when someone recants if they have already admitted to using the drug on at least two previous surveys. (Which would be administered at least two years apart.)

What they found was that the people most likely to recant were the ones who got jobs where they would now be fired for admitting drug use. For example, cops recanted drug use after previously admitting to it at twice the rate of the general public. So that's how they made the decision to discount recanting for people who had already admitted to drug use twice or more times.

As for people who try to lie and claim they are taking drugs that they haven't been, this is corrected for by asking about the same drug multiple times in different ways. I think they may also salt the questions by asking about drugs that don't exist.

You can obviously read all about the methodology of Monitoring the Future by downloading it, and here is an enormous PDF with tons of studies on the validity of self-reported drug use in general:

http://archives.drugabuse.gov/pdf/monographs/monograph167/do...

All in all the methods they are using seem fairly accurate. It would definitely be a little higher if they corrected for more stuff, so for example when they say that 82% of Americans have used marijuana, it's actually probably closer to 85 or 86% when you correct for additional things that they haven't.

Re:Poll: How old are you? by lukeschlather on Friday February 04, 2011 @01:01AM (#2178556)
I really don't think you can adjust away the teenage propensity to lie on these sorts of things. If there's one thing students in the US are taught effectively, it's how to game multiple choice tests. Throwing in trick questions isn't going to do much, especially for your AP-track students.
Re:Poll: How old are you? by Alex3917 on Friday February 04, 2011 @09:32AM (#2179497)
"I really don't think you can adjust away the teenage propensity to lie on these sorts of things."

The research seems to show that the self-reported responses are pretty accurate. For example, the number of people who reported using LSD basically went to zero right after the Pickard bust in 2003, when pretty much the entire US production got shut down. This means that if there are people lying about taking drugs they haven't been taking, it can't really be much than a couple tenths of a percent.

Re:Poll: How old are you? by timr on Thursday February 03, 2011 @06:11PM (#2177420)
Well, as I said before -- there are lots of caveats. But I hope that professor loses his bid to introduce another sound-bite into the statistics world, because it's not correct to blindly say that the data are "worthless" simply because they're voluntary responses. We make useful inferences from voluntary response data all the time (c.f. every political poll ever conducted, where people can choose not to respond). The correct question is: does the voluntary response process result in a systematic bias that will affect the conclusion in question? But that's neither short, nor catchy.

In this case, there's a strong argument to be made that the samples could skew young (if for no other reason than because younger people frequent sites like HN more than older people). It's also quite possible that the demographics for HN and for slashdot are different in a way that would make the HN audience skew younger. But the voluntary response process itself? I think it's a stretch to claim that it'll cause a measurable bias in age distributions, across two different surveys, spaced a decade apart. You'd have to argue that younger people in 2011 are more likely to answer a poll than younger people in 1998 were to answer a poll. Possible? Sure. The most likely source of error here? Nah.

In other words, if you're going to pick (and you should!), pick on the BIG, OBVIOUS differences between the data sets...not the little ones.

Re:Poll: How old are you? by jkent on Thursday February 03, 2011 @06:26PM (#2177481)
Do you have reason to doubt the validity of this particular survey?

I personally can't see significant problems of this over any survey technique.

There are problems with questionnaires and self-responses, sure, but to say they are worthless is to exclude an entire category of data based on a selective opinion. Horses for courses, if you like.

Many statistical methods are available, some of which are more appropriate than others in particular situations. There is little wrong with convenience sampling if what is of primary importance is convenience, e.g.

(That being said, I'd be careful with taking the results of this poll and applying it to any technology site and therefore the interwebs etc. Inferring from this data that we don't have an ageing population problem would be dangerous).

Re:Poll: How old are you? by petercooper on Thursday February 03, 2011 @08:22PM (#2177885)
[..] is nothing like you'd expect if everyone who started in their 20s in 1998 stuck around until today.

I have an interesting observation around this. I got started in this industry "for real" in the late 90s and worked for some "new media" companies (as they were known at the time!). I also read a lot of the magazines that featured things like "The 50 Most Important People Online" and similar crud.

And.. recently I tried tracking down people whose names I could remember back then. Few were still involved with tech or Internet-specific work. And of those "important" people, few were names I recognized.

To me, at least, it seems like the group of people working online has changed a great deal in the last 13 years. And I'm surprised by that.

Re:Poll: How old are you? by tungwaiyip on Thursday February 03, 2011 @05:23PM (#2177203)
It was same group of people responding. They have grown old :-O