News for hackers. Stuff that matters.
Re:Keeping computers from ending science's reproducibility by jerf on Friday January 22, 2010 @03:07PM (#1070543)
"Weather simulations have even more data; they are less accurate for other reasons."

I doubt anyone can beat particle physics for sheer information quantity. I seriously doubt that we have petabytes of real weather information to feed our simulations. I can find some references online to petabyte stores for weather simulation results online, but that's ultimately just cache, not information (in the information theoretic sense).

This is part of what I mean when I talk about the information theory, and how you can't get more information in the information-theoretic sense than the sum total of the simulation and the original data. Weather simulations may chew through terabytes or petabytes of RAM in the simulation phase, but they are not fed that much data. If the people involved mistake it for real information, then this is also part of what I mean when I say that once you get into using computers in a big way my training does indeed start giving me standing to complain again by even the most rigid "stay out of my science" standards.

Secondly, your assumption that running the simulation again is useless in a world where you can casually assume that you have all the data and the simulation and can have the contempt bred of familiarity for the whole process. In the real world, if you can't replicate the results, how can you criticize the model? I think there's still some "the computer said it, it must be right" underlying your answer; you can't assume the computer model is worth anything, it must be proved and debated and peer reviewed, which is not possible if you can't even get it to run and get the same results. The model is still subject to scientific inquiry, it can't be given a free pass.

Independent replication of results is also desirable, but you need both. In a world where nobody can replicate the results and it's hard to verify the simulation against the real world, it's too damned easy to end up with the Feynman electron mass situation where the selection effects from the researchers dominate the putative results of the simulations. The researcher summaries of the results of some runs of some models you can't see or execute and some data you can't get at happen to align... what does it mean? Frankly, who knows?

Re:Keeping computers from ending science's reproducibility by Retric on Friday January 22, 2010 @03:52PM (#1070657)
Several satellites take high resolution real time pictures of global weather they run 24 x 7 for years and that's RAW data. There are several of these plus radar stations etc. NOAA uses a subset of that information to make weather forecast (they normally toss out old data and data from the other side of the planet because it's not useful even if it might make a slightly better forecast). We can make highly accurate block by block forecast an hour ahead over major cities and just about any point in the US. However, there is little point in reading that level of detail from a forecast for such a short period of time. Sometimes when the forecast is 50% chance of rain over the next 8 - 12 hours they know where and when it's going to rain they just don't know where you are.

edit: One of the world's largest scientific data systems, NASA's Earth observing system data and information system (EOSDIS) has stored over three petabytes of earth science data in a geographically distributed mass storage system. that's just Nasa and a lot of their data does not make it into EOSDIS.

The goal of science is to understand the world. Running the same simulation on the same data and getting the same result only tells you that the machine running the simulation is not broken. What you want is to run a different program with different assumptions on different data and come to the same conclusion. This is actually used to make 7 day forecasts. They run a few different models with different assumptions and pick the average result. Over time each model is updated independently to maintain its independence.

PS: You don’t validate E=MC^2 by doing the exact same experiment 10,000 times. You do every type of experiment that you can think of which relates to E=MC^2 looking for anything which does not work out the way you think it should.

Re:Keeping computers from ending science's reproducibility by scott_s on Friday January 22, 2010 @04:37PM (#1070749)
I think this ongoing discussion is fascinating, and I don't disagree with you.

But (and you knew that was coming), re-running someone else's code, or reproducing someone else's experiment is verification. It doesn't mean that what they did is valid, but it verifies that they did do what they said they did.