News for hackers. Stuff that matters.
Re:Challenge to scientists: does your ten-year-old code stil... by sitkack on Monday August 24, 2020 @01:30PM (#24263042)
> exact worry people have when it comes to releasing code: people claiming that their non-software engineering grade code invalidates the results of their study.

If code is what is substantiating a scientific claim, then code needs to stand up to scientific scrutiny. This is how science is done.

I came from physics, but systems and computer engineering was always an interest of mine, even before physics, I thought it was kooky-dooks that CS people can release papers w/o code, fine if the paper contains all the proofs but otherwise it shouldn't even be looked at. PoS (proof-of-science) or GTFO.

We are the point in human and scientific civilization that knowledge needs to prove itself correct. Papers should be self contained execution environments that generate PDFs and resulting datasets. The code doesn't need to be pretty, or robust, but it needs to be sealed inside of a container so that it can be re-run, re-validated and someone else can confirm the result X years from now. And it isn't about trusting or not trusting the researcher, we need to fundamentally trust the results.

Re:Challenge to scientists: does your ten-year-old code stil... by snowwrestler on Monday August 24, 2020 @05:19PM (#24265653)
The history of physics is full of complex, one-off custom hardware. Reviewers have not been expected to take the full technical specs and actually build and run the exact same hardware, just to verify correctness for publication.

I doubt any physicist believes we need to get the Tevatron running again just to check decade-old measurements of the top quark. I don't understand why decade-old scientific software code must meet that bar.

Re:Challenge to scientists: does your ten-year-old code stil... by martingab on Wednesday August 26, 2020 @10:32AM (#24282820)
They didn't rebuild the Tevatron but still were able to rediscover the top within a different experimental environment (i.e. LHC with tons of different discovery channels) and have lots of fits for it properties from indirect measurements (LEP, Belle). Physics is not an exact science. If you have only one measurement (no matter if its software- or hardware-based), no serious physicist would fully trust in the result as long as it wasn't confirmed by an independent research group (by doing more than just rebuilding/copying the initial experiment but maybe using slightly different approximations or different models/techniques). I'm not so much in computer science, but I guess here it might be a bit different ones a prove is based on rigorous math. However even if so, I guess, it's sometimes questionable if the prove is applicable to real-world systems and then one might be in a similar situation.

Anyways, in physics they always require several experimental proves for our theory. They also have several "software experiments" for e.g. predicting the same observables. Therefore, researchers need to be able to compile and run the code of their competitors in order to compare and verify the results in detail. In this place, bug-hunting/fixing is sometimes also taking place - of course. So applying the articles suggestions would have the potential to accelerate scientific collaboration.

btw; I know some people who do still work with the data taken at the LEP experiment which was shut down almost 20 (!) years ago and they have a hard time in combining old detector-simulations, monte-carlos etc. with new data-analysis techniques for the exact same reasons mentioned in the article. For large-scale experiments it is a serious problem which nowadays has much more attention than at LEP ages, since LHC has anyways obvious big-data problems to solve before their next upgrade, including also software-solutions.

Re:Challenge to scientists: does your ten-year-old code stil... by reitzensteinm on Monday August 24, 2020 @10:56PM (#24267757)
If you could have spun up a Tevatron at will for $10, would the culture be the same today?

I suspect that software really is different in this way, and treating it like it's complex, one off hardware is cultural inertia that's going to fade away.

Re:Challenge to scientists: does your ten-year-old code stil... by matthewdgreen on Monday August 24, 2020 @01:42PM (#24263187)
All of my 2010 scientific code runs on the then-current edition of Docker. /s
Re:Challenge to scientists: does your ten-year-old code stil... by sitkack on Monday August 24, 2020 @01:51PM (#24263304)
I made no mention of Docker, VMs or any virtualization system. Those would be an implementation detail and would obviously change over time.

A container can be a .tar.gz, a zip or a disk image of artifacts, code, data and downstream deps. The generic word has been co-opted to mean a specific thing which is very unfortunate.

Re:Challenge to scientists: does your ten-year-old code stil... by matthewdgreen on Monday August 24, 2020 @02:15PM (#24263586)
My point, which I guess I did not make clearly enough, is that container systems don't necessarily exist or remain supported over the ten-year period being discussed. The idea of ironing over long-term compatibility issues using a container environment seems like a great one! (For the record, .tgz -- the "standard" format for scientific code releases in 2010, does not solve these problems at all.)

But the "implementation detail" of which container format you use, and whether it will still be supported in 10 years, is not an implementation detail at all -- since this will determine whether containerization actually solves the problem of helping your code run a decade later. This gets worse as the number, complexity and of container formats expands.

Of course if what you mean is that researchers should provide perpetual maintenance for their older code packages, moving them from one obsolete platform to a more recent one, then you're making a totally different and very expensive suggestion.

Re:Challenge to scientists: does your ten-year-old code stil... by sitkack on Monday August 24, 2020 @11:32PM (#24267918)
Of course of course. I am not trying to boil the ocean here, or we would have a VM like wasm and a execution env like wasi and run all our 1000 year code inside of that.

The first step is just having your code, data and deps in an archive. Depending on the project and the age, more stuff makes it into the archive. I have been on projects where the source to the compiler toolchain was checked into the source repo and the first step was to boostrap the tooling (from a compiler binary checked into the repo).

We aren't even to the .tar.gz stage yet.