Re:Science Fairs Aren't So Fair
by jerf on Sunday March 15, 2015 @06:18PM
(#9208450)
"Also a stealth observation is the hidden assumption that a negative or null result means the experiment was a failure."
I want to say that the "science" of the science fair is a sick parody of the real process of science, except the sick parody is itself now the real process.
One of the few things from grad school that still can cause me to momentarily crinkle my eyebrows in annoyance 12 years later is when I was in my machine learning class, and we were told to go use some automated decision tree learning software to go learn something from some data set. I deliberately chose a dataset and question that should have produced a null result, precisely to validate that the technique in question did in fact produce a null result. Despite the fact the professor completely agreed that I correctly converted the data, correctly formulated a hypothesis, correctly tested the hypothesis, and correctly determined the null hypothesis could not be discarded... I took a 50% deduction on the assignment. A thing that was ultimately irrelevant since I still got the max final grade in the course... what furrows my brow isn't the actual bad grade, but what it says about the mentality of what science is, even at that high level.
If that still doesn't sound scandalous to you... and it doesn't sound scandalous to a lot of people at first... consider the outcome if I had not chosen a data set that I expected to be null. Suppose I chose a data set that I expected to be able to learn something from, and I was wrong. (This happens a lot in the real world with real data, after all!) What does this mean? It means that 50% of my grade would have been dependent on me massaging the data until I did get a result. Yow! Real science should not be encouraging people to mine data until they get results like that. And one need not spend very long looking at how real science is done to see this idea is quite pervasive in "real science".
Real science is not compatible with the idea that a non-null result is "success" and a null result is "failure". Success and failure have to do with whether the result, whichever of those things it may be, correlates with reality. Ask a scientist this question directly and there's at least a decent chance they'll mouth words about how important the reality correlation is, but look to their actions and it's all "result = success".
I'm gonna be brutally honest here... my plan is that when the kids hit the science fair age, I'm going to tell them to just do what they need to do to get by, and explain to them what real science is. The two things are in direct conflict, and I'll take real science any day.
I want to say that the "science" of the science fair is a sick parody of the real process of science, except the sick parody is itself now the real process.
One of the few things from grad school that still can cause me to momentarily crinkle my eyebrows in annoyance 12 years later is when I was in my machine learning class, and we were told to go use some automated decision tree learning software to go learn something from some data set. I deliberately chose a dataset and question that should have produced a null result, precisely to validate that the technique in question did in fact produce a null result. Despite the fact the professor completely agreed that I correctly converted the data, correctly formulated a hypothesis, correctly tested the hypothesis, and correctly determined the null hypothesis could not be discarded... I took a 50% deduction on the assignment. A thing that was ultimately irrelevant since I still got the max final grade in the course... what furrows my brow isn't the actual bad grade, but what it says about the mentality of what science is, even at that high level.
If that still doesn't sound scandalous to you... and it doesn't sound scandalous to a lot of people at first... consider the outcome if I had not chosen a data set that I expected to be null. Suppose I chose a data set that I expected to be able to learn something from, and I was wrong. (This happens a lot in the real world with real data, after all!) What does this mean? It means that 50% of my grade would have been dependent on me massaging the data until I did get a result. Yow! Real science should not be encouraging people to mine data until they get results like that. And one need not spend very long looking at how real science is done to see this idea is quite pervasive in "real science".
Real science is not compatible with the idea that a non-null result is "success" and a null result is "failure". Success and failure have to do with whether the result, whichever of those things it may be, correlates with reality. Ask a scientist this question directly and there's at least a decent chance they'll mouth words about how important the reality correlation is, but look to their actions and it's all "result = success".
I'm gonna be brutally honest here... my plan is that when the kids hit the science fair age, I'm going to tell them to just do what they need to do to get by, and explain to them what real science is. The two things are in direct conflict, and I'll take real science any day.