News for hackers. Stuff that matters.

A comment on 1 in 4 Statisticians Say They Were Asked to Commit Scientific Fraud. Parent comment

Re:1 in 4 Statisticians Say They Were Asked to Commit Scient... by aluren on Friday November 02, 2018 @06:58PM (#18367411)
It's not at all uncommon that implementations provided with papers do not actually do what the paper says it does. You're often lucky when there is an implementation at all. But most of the time it's just that running the exact same implementation under the same conditions requires setting up a very specific environment, installing specific versions of libraries, using some niche software and converting data from one byzantine format to another. Each deviation from the original paper is liable to subtly affect the results in nondeterministic ways. That's why no one really gets surprised or even cares, life is too short to call out all the bad software written in academia.
[ Parent | on HN ]
Re:1 in 4 Statisticians Say They Were Asked to Commit Scient... by xorcist on Saturday November 03, 2018 @06:46AM (#18369585)
If a research result requires specific versions of libraries to replicate, wouldn't that completely invalidate any scientific claim being made?

Any change in precision or numerical methods that affects results surely must be well within the error margins.

[ Parent | on HN ]
Re:1 in 4 Statisticians Say They Were Asked to Commit Scient... by aluren on Saturday November 03, 2018 @07:40AM (#18369745)
I was more referring to changes in the API where the input and output suddenly have to be in different formats in the middle of a pipeline, causing a crash. What can also happen is that somehow the old format is still valid and gets processed all the same, thus yielding nonsensical results. Sometimes a lab devises their own format which no one else uses, and the specification may be updated without notice between the moment they publish and the moment you try things out. Most people have no idea about things like 'backwards compatibility', 'unit tests', 'containers', etc. Code is just a tool to them and the fact that they had to write some is annoying them in itself.
[ Parent | on HN ]
Re:1 in 4 Statisticians Say They Were Asked to Commit Scient... by RidingPegasus on Friday November 02, 2018 @09:09PM (#18368048)
> requires setting up a very specific environment, installing specific versions of libraries, using some niche software and converting data from one byzantine format to another

Containerisation is fairly mature and simple to use. Many in other fields struggle with these exact same issues and are able to create reproducible environments just fine.

I find it amazing that those publishing don't include their implementation, all that work locked away on a rusty hdd.

[ Parent | on HN ]
Re:1 in 4 Statisticians Say They Were Asked to Commit Scient... by Fomite on Saturday November 03, 2018 @02:56AM (#18369077)
Containerization is not simple to use for many users of scientific code.

Hell, I'm pretty sure in my department, the first response would be "What's that?"

[ Parent | on HN ]
Re:1 in 4 Statisticians Say They Were Asked to Commit Scient... by thomasz on Saturday November 03, 2018 @03:53AM (#18369198)
Containers may be a bit difficult, but every researcher should be capable of using a VM and distributing it.
[ Parent | on HN ]
Re:1 in 4 Statisticians Say They Were Asked to Commit Scient... by Fomite on Saturday November 03, 2018 @04:01AM (#18369209)
I'm excited if I can get my colleagues to use version control.
[ Parent | on HN ]
Re:1 in 4 Statisticians Say They Were Asked to Commit Scient... by gpm on Saturday November 03, 2018 @11:05AM (#18370527)
Do you think that they would use a VCS that was less invasive and more transparent to their workflow, like dropbox?

I'm thinking of making a VCS that simply runs in the background and

- Automatically records every file save (effectively a git commit without a message)

- Allows adding messages through tagging (like git tag)

- Handles 'branching' just by asking you make a copy of the directory with a different name, properly understands how to diff/merge/etc copied files/directories that have since diverged.

- Has built in support for large files

[ Parent | on HN ]
Re:1 in 4 Statisticians Say They Were Asked to Commit Scient... by Fomite on Saturday November 03, 2018 @06:13PM (#18372781)
Almost certainly, yes.
[ Parent | on HN ]
Re:1 in 4 Statisticians Say They Were Asked to Commit Scient... by thomasz on Saturday November 03, 2018 @09:10AM (#18370012)
Git is way more complicated than VMware.
[ Parent | on HN ]
Re:1 in 4 Statisticians Say They Were Asked to Commit Scient... by user5994461 on Saturday November 03, 2018 @10:16AM (#18370237)
Use SVN just like everybody else.
[ Parent | on HN ]
Re:1 in 4 Statisticians Say They Were Asked to Commit Scient... by umanwizard on Saturday November 03, 2018 @03:53PM (#18372123)
You obviously don't know many researchers (if any).
[ Parent | on HN ]
Re:1 in 4 Statisticians Say They Were Asked to Commit Scient... by sk5t on Saturday November 03, 2018 @12:17AM (#18368662)
In the software dev world--but outside of BSD--I would argue that containerization is extremely immature. Outside the software dev world, it is not easy to use.
[ Parent | on HN ]