The bottom line? Studies using a group of undergraduates on a day-long project are worthless.
The other thing I have seen are efforts to measure professional programmers via time sheets, even had to do it myself, usually in the guise of getting us to produce more reliable estimates. Not more accurate estimates, but more reliable in the sense that Jack is always off by 4x while Jill is consistently off by 1.5x. These efforts usually died because programmers resisted the hassle.
Other measurements have been equally worthless. Measure by lines of code? You get lots of blank line and comments. I have even worked on projects that required 2 pages of comment for 5 lines of code.
PS: I have completed the same task as other programmers in less than 10% of the time. However, assessing overall productivity requires that someone is consistently faster on a wide range of tasks not just that they happen to solve something in 10% or even 1% of the time.
I've never taught a CS course, but back in my college days I helped a friend with his CS homework by taking advantage of weak file permissions to copy other people's homework. The shocking thing was that, looking at a small sample, most of the programs didn't work correctly. To complete the assignment our way, we had to fix bugs in programs that other people wrote, which is fantastic preparation for a professional programmer ;-)
I don't think the concept of productivity can span multiple fields. Does a C programmer have the same productivity as a Ruby programmer? On the other hand, maybe some part of a Ruby project needs optimizing, so a routine is written in C. Maybe there can be a definition of productivity that crosses languages... but we don't have one yet.
So I would choose to politely disagree. You can measure productivity.
Am I negatively productive under your measures?
1. Familiarity with the library set / language
2. Familiarity with the problem space
3. Familiarity and speed with tools/programming environment
4. Research speed (and research tools/materials)
5. Mental Program structural planning speed
6. Previous Experience with the specific problem
7. Knowledge with time saving programming structures
8. Skill with mathematics and algorithms
9. Computer Hardware & Internet Connections
10. Ability to choose the best tools for the job
11. Learning speed
12. Motivation, energy, ability to concentrate for long periods of time
And that's listing mostly external factors that are somewhat obvious, not some less obvious speed of mental thinking or something similar.
You can define developer productivity as speed to complete an identical project compared to another and the quality/maintainability/readability of the software they produce to match the project requirements.
Now back to Terminal and vi so I can be productive again...
Part of my point was, though, that nobody should need to do an objective study. :) Because it's pretty frickin obvious to anybody with a brain who works in the field, in my judgment. And it's not limited to our field. It's a human-wide phenomenon.
Anyways, it's really an unfalsifiable claim, isn't it? For any study that doesn't show a 10x difference, you could claim that it didn't have any truly "good" programmers.
The burden of proof should be on those who make the claim.
"It's important to note that (as far as I can tell) the author doesn't actually debunk the existence of precognition, only the claim that studies show precognition to exist."
Yes, but quite vacuous. The author can't prove the non-existence of evidence supporting the 10x claim. He was critiquing our folklor-ish belief in this 10x claim, without evidence, and our gross indifference to this fact.
Let my try another tack, proposing: excluding some bottom <10% that are incompetent, and those with less than 5 years of experience, all programmers have exactly the same 1x productivity capability; all apparent levels of difference are really either ramp-up on new technologies or flaws in development processes and tools. <citation of irrelevant study on something else> <citation of myself, citing the aformentioned study>.
... do you now believe in this 1x claim? No? Hmm, perhaps I need to get it quoted more and then you'll believe it?
The author's point is that "quoting it more" and "it seems reasonable" are the only basis for our belief in the 10x claim. But neither are scientifically, empirically relevant. We should follow up with further studies to confirm the effect, and then investigate the cause. But we don't. And because we have not, we don't have any scientific basis for believing this 10x claim.
This isn't a matter of "dismissing all of our research based on it" -- there is nothing to dismiss. There has been no sufficient research.
Oddly, my initial misread is inaccurate. There are far more studies supporting the existance of precognition than supporting this 10x claim. ;-)
indeed! computer science is a social science! http://www.achangeiscoming.net/docs/cssocsci.html
Personally, I think the author is unintentionally also pointing out the problem of relying on studies of programmer productivity: it's next to impossible to measure. I seriously doubt that there is no valid research on this because no one has gotten around to it. Too many peoples' businesses rely on understanding programmer productivity inside and out for that to be the case.
Personally, I agree with whoever it was that argued that we need to view this as a soft science like psychology or sociology. We need to focus on working in spite of our imperfect means of measuring productivity rather than dismissing all of our research based on it. Such is the nature of studying the human mind.