I fear this contradicts your assertion that published results are not garbage. If researchers are claiming their systems are strong based on the fact that they beat weak baselines, then their claims are rubbish and their scholarship shoddy. If, on top of that, their work is difficult to reproduce, or replicate (by the distinction in the paper) then the published results are beyond garbage- they are meaningles garbage, garbage that fails to prove even its weak and insignificant claim to beat a weak baseline.
Basically, this is exactly the kind of shoddy scholarship that Backus accused social scientists of, in his infamous speech about "Cargo-Cult Science". In that speech, he points out that basing new research on previous weak research can only lead in the proliferation of weak results (my interpretation of his words, from memory).
This seems very much to be the case in machine learning research today. This article only provides further evidence that there is something rotten to the core in machine learning research.
>> Reviewers should push back against weak baselines, but be more accepting of results that make modest claims.
Agreed about the second part, however, the paper in the OP says that the way things are, it is very difficult to know which baselines are strong. My interpretation: because they're all trying to beat weak baselines that became established by beating earlier weak baselines. It's weak baselines all the way down.
Also, sorry HN, for the controversial comment (judging by the rollercoaster vote activity). I hope it adds something to the discussion despite the tone and I'll remember to keep the tone down in the future (but I keep saying that and I always end up giving in to the temptation to use more colourful language. I suck and I'm sorry).
It makes pulling out the actual content of the papers such a slog.
[0] exhibit A: https://ieeexplore.ieee.org/document/1017616
Source: am former producer/publisher of abomination research code.
Months of tweaking code is exactly what you want some test coverage for - so that you end up with a tweaked version of something where you still know what it does instead of something that does god knows what.
Edit: Perhaps one way to encourage this is via topic "reuse" papers, which present further detail on a previously published papers topic, but this time with code and more detailed discussion. It would at least help the situations in that the author gets some reuse of effort, but in a way that still shows new information and advances the field.
Literate programming suffers from this in general because you don't want to clutter up your document with noise about versions and so forth - even as appendices, including package information and build instructions is a lot of noise.
Requiring that papers making claims about some code base publish that code base simultaneously with the paper via the peer review process should be sufficient to improve the overall state of affairs.
Don't underestimate the value of demos and competitions with companion papers either, those tend to get a lot more notice.
In general, you'll have a lot of approaches that don't work out which are nice to have a record of, but definitely don't merit being in the final paper.
If your claims don't depend on specific behaviours of some body of code, you wouldn't need it - eg, an article claiming some asymptotic performance of an algorithm should describe the algorithm in the abstract s.t. the performance bound can be proven, not some specific language's implementation of the algorithm.
distill.pub has a journal that strongly encourages interactive articles: https://distill.pub/journal/
It's a lot of work, tho:
> One good test for whether your article is a fit for Distill is whether your collaborators and you are willing to put in whatever time is necessary to write and illustrate an outstanding article. In our experience, this often takes 100+ hours.
But for most journals, it's the story as usualy. Personally I dont feel very confident about a paper result if they don't publish the code. Why hold back on the code?
I know you know this, but this is absolutely a feature.
http://matt.might.net/articles/crapl
> The CRAPL is an open source "license" for academics that encourages code-sharing, regardless of how much how much Red Bull and coffee went into its production.
The big question is why this failure to publish code is tolerated by program+steering committees.
Besides, _ideally_, machine learning reseach is not about building a system but about making a theoretical claim that is supported by empirical results. The main claim should be about a property of the algorithm: its time complexity, its convergence, its correctness, etc. Experiments should directly test the theoretical claim and verify it in practice.
... ideally. In practice most machine learning papers are vehicles for a little table that compares your system with a baseline that you chose based on the fact that your system beats it (or you wouldn't have put it in the paper). Whether your experiments prove anything beyond the fact that you beat some benchmark is not really important.
And so whether someone publishes a reproducible paper is of no consequence because the only point of papers is to pass review and convince someone that the researchers should stay in the (research) money. In any case, things move so fast that most of the time nobody wants to reproduce your old and unsexy paper from last week.
Then some busybody comes along and points out the fact that you're wearing no clothes. Like in the paper above.
Also, more likely and non- nefarious, but no less harmful, there's just an academic culture that says "this is what a paper looks like and this is what it contains". Moving from this pattern increases the odds you'll not be published, and it would also likely result in a scandal when someone uses it to point out that the peers in peer review are neither peers (they can't read nor understand nor test the code) nor reviewing (they won't actually put resources into code review).
As a non-wealthy, non-academic, this isn't a battle I'm interested in fighting directly. There are other hills I'd rather die on :)
In some cases I basically had to run a test harness and brute force the parameters they'd used, because the algorithm was basically useless without picking parameters in the right range.
It was hard to accept then (~2008-ish), but now we're at a point where frankly at least for the more basic CS papers we ought to be able to expect source and some runnable format (e.g. a VM image) with test data.
I handed in a DVD with gigabytes of test data, full source, built binaries, and a test harness for the work I did for my thesis. All it took was to follow practices that I needed to follow to satisfy myself that my results were valid anyway.
I realize that for some work that relies on much larger and/or proprietary datasets making everything available would be hard, but a small test case illustrating the code run with the right parameters etc. ought to be a minimum unless there are substantial barriers, in which case the paper ought to signpost clearly what is missing and why.
The paragraphs of English text help them to be able to claim that they are indeed the authors of some novel method (exactly what the author wants).
Code/data accessibility helps others to both find errors/imperfections in their approach and to build upon their results (exactly what the author doesn't want).
I'm out in the trenches trying to turn ding dong papers into tools that does something useful, and the "muh mape 50% better" papers are effectively always something that should be ignored. Actually better algorithms and implementations (something like gradient boost machines/xgboost) are rare enough attempting to find them in the literature is futile.
Lack of availability of data and code should be an immediate disqualifier from publication for research. In any field. There is no substantial difference between withholding source code or data and withholding experimental procedures or list of materials tested. I can only think that the people who withhold such things are either actively malicious or are completely ignorant of the stupendous danger of cutting corners in science. Given the history we have to work with, it's pretty clear that the distance from cutting a corner in a way that seems harmless to the deaths of millions of people is very short. This is not something people should trifle with.
“What gets me is the thought that even trying our best to figure out what's
true, using our most sophisticated and rigorous mathematical tools: peer review,
and the standards of practice, we still get it wrong so often; so how frequently
do we delude ourselves when we're not using the scientific method? As flawed as
our science may be, it is far away more reliable than any other way of knowing
that we have.”
What do we have now:
1. We have a common platform, which is a de-facto standard for sharing (somewhat math-related) papers: arxiv.org. Basically, anyone can upload anything to arxiv.org, there's not much of quality review required. Which, all in all, I think is great. I mean, it's orders of magnitude better than other fields, where both publishing and accessing a published paper is expensive, painstakingly slow and inconvenient process which is obviously great for the journals that have a monopoly over it, but less so for everyone else. All being justified by the need of peer-review (which is not even sure to be working alright, because (1) we all know the quality of the vast number of studies in some social field is shit, and (2) you can even publish a completely bullshit paper in a very reputable journal if you try hard enough).
2. Much of the research is done in the academia (because, well, that's supposedly the reason it exists), but, paradoxically, the main moving force of the progress isn't really interested in progress. It is a competitive environment, so everybody is interested in publishing a paper which will be recognized as an "important one" later on, so the. They are not interested in anyone learning anything and especially not interested in making their rivals (and that is what any other researcher is, a rival) able to make a follow-up contribution easily.
3. Hence, many important studies by reputable scientists lack quality, which everyone knows, including the scientists. Papers are written in a passive-agressive tone, in particularly confusing terms an inconvenient format, they often lack the source code and the source data, a minor one-sentence contribution to the current state-of-the-art is spread around 20 pages which you have to read fully to understand what exactly the contribution is.
4. To summarize: everyone is interested in publicity of their work, nobody is interested in improving their work because it provides them the high ground, but everyone (including even those outside of the academia) is interested in improving others work, since it helps them to learn. Improvements on the others work, obviously, are not shared unless it can be framed as a follow-up study.
How (I think) it might be improved. Assume a platform everyone uses to share their work, let's call it arxiv2. arxiv2 has a ranking system, which pushes the papers it "thinks" are of the better quality to the top. "Quality" is (maybe not exhaustively, but that's the main idea) a set of domain-specific flags which are considered to be beneficial for a study to be conclusive in this field, like "has a source code", "has a source data", "has a benchmark/accuracy/specificity/precision metrics", "summary/abstract succinctly captures the main result". Depending on the field, features may be more complicated in order to address some common problems like p-hacking or non-representative sampling in the study. But all the metrics have to be concrete. Nobody is required to include anything, but if they don't, and if others mark the work as lacking some features, it has less chance to get to the top. Everyone can vote for the paper, some minimal moderation may take place. Disputes may be solved similarly to how they are solved on wikipedia/stackoverflow (which are, admittedly, awful and frustrating, but order of magnitudes better than the status-quo in the research).
Does this mean that the published results are garbage? No! But what does it mean?
First, don't miss the main claim: less than half the published papers have sufficient code and data available for replication. While one might argue exactly what the standards for availability should be, if journals and reviewers were to demand it, this number can definitely be improved. Greater availability of data and code makes for better science.
Second, my guess is that it means that paper authors are using excess "puffery" in an attempt to improve the chances that their papers are accepted for publication. They've decided that using a weak baseline and claiming a 50% improvement is more likely to result in publication than using a realistic baseline and claiming a 0-5% improvement. Unfortunately, they are probably right.
The useful change (in my opinion) would be to change the standards by which papers are judged to be publishable. Reviewers should push back against weak baselines, but be more accepting of results that make modest claims. You can still have a useful paper even if the degree of improvement is small --- or even nonexistent. Maybe it will inspire someone else, maybe it's useful in an ensemble, or maybe it applies to a case where the baseline wouldn't. It's the puffery that's the problem, not the publication. Creating a system that rewards authors who more honestly evaluate their work would be an overall win.