News for hackers. Stuff that matters.
Re:Programmers Need To Learn Statistics Or I Will Kill Them ... by dschobel on Tuesday May 26, 2009 @11:57AM (#626897)
Serious question: who here who has at least a BS in CS didn't have a mandatory stats class where you learned all about picking sample sizes to give you an accuracy you're happy with or that an average without a standard deviation is nigh-meaningless?

It's just a fact of our profession, there is a significant percentage of people who just slap together APIs and have zero understanding of the maths behind it.

I don't see why it takes Zed 1000 words to say it or why he has to get sanctimonious about it.

Re:Programmers Need To Learn Statistics Or I Will Kill Them ... by megaduck on Tuesday May 26, 2009 @12:59PM (#626993)
The problem isn't that people know nothing about statistics. The problem is that they think they know more than they actually do. It's fairly easy to get past your "intro to stats" class with only a passing familiarity with the subject and no deep understanding. Only once you truly grasp stats do you realize that you don't actually know anything.

Any statistical formula will predict its own failure. That's kind of mind-blowing, and the implications aren't always grasped after just one semester (or quarter) studying the topic.

Re:Programmers Need To Learn Statistics Or I Will Kill Them ... by m_ on Tuesday May 26, 2009 @01:15PM (#627019)
I studied CS and I got past my "intro to stats" class with only a passing familiarity with the subject and no deep understanding (aside from the probability aspects of the course).

The professor was very poor overall. I was able to cram enough to do well, but I retained very little after the final. I would love a deep understanding built from the foundations-- the coursework we had was a lot more obtuse and "take my word for it".

In a sense, I felt I got a better foundation for statistics from a freshman sociology class than from the Math professor.

Re:Programmers Need To Learn Statistics Or I Will Kill Them ... by stcredzero on Tuesday May 26, 2009 @02:18PM (#627163)
Why aren't there statistics for CS classes that put stats primarily in the context of system profiling and metrics? Are there any out there?
Re:Programmers Need To Learn Statistics Or I Will Kill Them ... by cschep on Tuesday May 26, 2009 @02:36PM (#627206)
That would have been cool. Stats wasn't required for us, but if it would have been Statistical Analysis of Computer Systems(or programs, or whatever), I would have signed up for it anyway.

I heard the stats class offered where I went to school was a joke so I didn't pursue it optionally. I'm sure it's different depending on your university.

Also, my school was not renowned for any sciences.

Re:Programmers Need To Learn Statistics Or I Will Kill Them ... by ZeroGravitas on Tuesday May 26, 2009 @03:41PM (#627364)
I once listened to a stats professor bemoaning that he had to teach stats to various groups of students in their first year when they didn't know enough about their own subjects for him to use any relevant examples.

Since stats on it's own can be a bit abstract that's a double whammy.

Re:Programmers Need To Learn Statistics Or I Will Kill Them ... by stcredzero on Tuesday May 26, 2009 @04:38PM (#627463)
Then stats in CS should be a 2nd or 3rd year course. By then, there should be a great deal of material usable as examples.
Re:Programmers Need To Learn Statistics Or I Will Kill Them ... by stcredzero on Tuesday May 26, 2009 @03:17PM (#627300)
A really good textbook along these lines might really be something.
Re:Programmers Need To Learn Statistics Or I Will Kill Them ... by mr_justin on Tuesday May 26, 2009 @05:07PM (#627514)
By "intro to stats" do you mean the calc-based stats class that most people in CS take? I'd hardly call that an intro class considering you need to complete Calc I & II before enrolling.
Re:Programmers Need To Learn Statistics Or I Will Kill Them ... by likpok on Wednesday May 27, 2009 @10:08AM (#628475)
Like many things, Stats is a vast field. Would you call a class about the basics of groups and rings anything other than an intro to abstract algebra, even though most places require both calculus and a fair amount of mathematical background?

This is sort of Zed's point. It is an intro class into a field with a great deal of depth. All you can do in one undergraduate semester is touch on a few of the most important parts.

Re:Programmers Need To Learn Statistics Or I Will Kill Them ... by baguasquirrel on Tuesday May 26, 2009 @02:17PM (#627161)
At CMU we had a mandatory stats class but it didn't have anything concerning sample size picking. Ironically, the stats class I had to take when I was in the humanities dept. was more comprehensive. When I walked into industry, I realized after two or three years that practically everyone I'd met knew jack shit about statistics. I myself learned most of what I know from humanities-major friends and books like NLP by Manning and Schutze.

Having been through what Zed's been through (on both the giving and receiving end to wit), I can empathize with why he'd get all sanctimonious about it, but I feel that it just makes the problem worse (at least in person). Programmers are already an egotistical lot, and I've learned that directly attacking their ego tends to make things worse.

What can be done then? I don't know. It feels like this is at the core a personality problem, in particular one in which people associate their ability to know with their sense of self, and personality problems are terribly hard to correct.

Re:Programmers Need To Learn Statistics Or I Will Kill Them ... by Periodic on Tuesday May 26, 2009 @06:10PM (#627623)
I'm currently doing a CS masters and I did my undergrad in Physics. I'm really glad that I have the background in mathematics and stats that I have. In our physics classes they really drilled into us that although our theory and homework often used exact numbers, no measurement meant anything without a measurement of the error. "A number without error is meaningless", they would say.

One thing I never fully understood was how to calculate the propagation of error. There are many useful tricks for reducing the impact of error on your final calculation, and a few things you need to watch to make sure it doesn't increase.

Re:Programmers Need To Learn Statistics Or I Will Kill Them ... by voberoi on Wednesday May 27, 2009 @04:29AM (#628193)
I didn't, and I'm graduating with a BS in CS from Stanford in a few weeks. We have an introductory stats requirement, but I found that the stats class I took my sophomore year for my (dropped) economics major was far more enlightening than my CS stats requirement.

I think the fundamental issue is that there's very little focus on statistical practice in most of these courses. Social scientists have it good: they're always taught how to deal with and interpret statistics in the same way they'll have to in their line of work. It's totally useless to throw a bunch of theory at students. I think that teaching statistics in the context of real problems is the only way (most) students will actually learn and come to appreciate how useful it is.

The department revamped the major this year, and they've introduced a mandatory stats class tailored to CS students: cs109.stanford.edu. I haven't looked through it at all, but I think it's a step in the right direction.

Finally, I have to give a shout-out to The Little Handbook of Statistical Practice (http://www.tufts.edu/~gdallal/LHSP.HTM) in this thread. It's an amazing resource for anyone who works with statistics. I've referenced it while doing performance testing, building an A/B testing system, and working on problem sets. From the website:

"My aim is to describe, for better or worse, what I do rather than simply present theory and methods as they appear in standard textbooks. This is about statistical practice--what happens when a statistician (me) deals with data on a daily basis."

Read it now!

Re:Programmers Need To Learn Statistics Or I Will Kill Them ... by lutorm on Wednesday May 27, 2009 @12:07PM (#628642)
For a different view of how useful statistical practice (when applied mindlessly) is, read "The Black Swan". It's all about how people use statistical models with Normal distributions in places where it's patently unjustified, and the price we pay for it.

I played this game myself with a friend. I sent him ten samples of a (for him) unknown distribution and asked him to estimate the mean. Then 100, then 1000. His estimate of the mean kept changing to higher and higher values, because the samples were drawn from a Pareto (power-law) distribution with a mean of 1000. Such a distribution is almost indistinguishable from one with a mean of infinity, because all the signal is in the very rare, large outliers. If you try to analyze samples from such a process assuming it's Gaussian, nothing will make sense, and the standard deviation will give you an estimated uncertainty of the mean that is far, far below the actual uncertainty.

Re:Programmers Need To Learn Statistics Or I Will Kill Them ... by menloparkbum on Tuesday May 26, 2009 @05:48PM (#627590)
Data point: I am about Zed's age and majored in math and was one class away from a CS double major and I didn't have to take any stats courses. When I was in school stats was a totally different track than what most math people took. For some reason biology and business majors had to take stats but physics and math majors did not. I feel like the physics/math/CS people assumed stats was easy and you'd pick it up as needed.
Re:Programmers Need To Learn Statistics Or I Will Kill Them ... by stcredzero on Tuesday May 26, 2009 @12:30PM (#626951)
There's a big percentage of programmers who don't understand O(n^2). On multiple occasions, I've seemed as a guru because of this one little tidbit.

What if the medical profession has as much egregious and widespread ignorance of the basics as programming? Would you be in favor of certification?

Re:Programmers Need To Learn Statistics Or I Will Kill Them ... by duncanj on Tuesday May 26, 2009 @01:50PM (#627099)
In medical professions, you go through a period of training (clinicals, internship, residency) where people with tons of experience point out that you never learned anything. This humbling experience is not really available in the CS world.
Re:Programmers Need To Learn Statistics Or I Will Kill Them ... by stcredzero on Tuesday May 26, 2009 @02:16PM (#627156)
Particularly disturbing, since the only good way I know of to learn programming and system architecture is through mentoring. Basically, our mentoring system is haphazard. There's even a lot of anti-mentoring happening out there.
Re:Programmers Need To Learn Statistics Or I Will Kill Them ... by SapphireSun on Wednesday May 27, 2009 @03:29AM (#628154)
I find this incredibly hard to believe. Anyone who has gone through a single intro course in CS should understand at least the difference between linear and quadratic time. Perhaps I've been in the university too long. Are these people non technical majors who picked up programming in PHP?
Re:Programmers Need To Learn Statistics Or I Will Kill Them ... by dschobel on Tuesday May 26, 2009 @01:01PM (#626998)
We have de facto certification. It's called moving to silicon valley :)
Re:Programmers Need To Learn Statistics Or I Will Kill Them ... by stcredzero on Tuesday May 26, 2009 @01:11PM (#627011)
And everywhere else, people have to put up with crappy programming? I think this system leaves a bit to be desired.
Re:Programmers Need To Learn Statistics Or I Will Kill Them ... by randome on Tuesday May 26, 2009 @05:18PM (#627537)
it seems pretty mind boggling that people are that bad at stats ... we have a mandatory stats class for CS ... it doesn't teach some of the stuff mentioned in the article but the stuff mentioned is completely intuitive -- you shouldn't need to take a stats class to understand that.

like if the variance of your result is really high and you're estimating the mean its clear that the number of samples you take is important !?!

Re:Programmers Need To Learn Statistics Or I Will Kill Them ... by megaduck on Tuesday May 26, 2009 @09:08PM (#627812)
You're proving his point, which is that people's intuition is often dangerously wrong. The sample size is actually often nowhere near as important as your sampling method. For an extreme example, I can conduct a billion coin flips with a weighted coin, but it's going to tell me jack about the general behavior of coin flips. Avoiding sample bias is a hard problem, far harder than most people appreciate.

To complicate matters further, "accuracy" is an extremely tricky concept in statistical analysis because error rates work very differently than they do in, say, physics. In physics, when you measure something you can be sure that your results are accurate, so long as you stay outside your instrument's range of error. In stats, your confidence interval just tells you how likely it is that your results are completely wrong, or even worse, wrong by a completely unknown amount. Every statistical inference you make has a chance of completely blowing up on you. That chance can be defined and reduced, but it can never be eliminated. There's also things like frequentist vs Bayesian statistics, where the interpretation of the same data can be completely different.

Zed may be a jerk sometimes, but on this topic he's dead right. Most programmers are far more confident about this stuff than they should be.

Re:Programmers Need To Learn Statistics Or I Will Kill Them ... by megaduck on Wednesday May 27, 2009 @12:29AM (#627986)
After doing some research, I think I've completely abused the notion of the confidence interval. Which I think just helps to prove Zed's point. This stuff be hard.