>> Second, my guess is that it means that paper authors are using excess "puffery" in an attempt to improve the chances that their papers are accepted for publication. They've decided that using a weak baseline and claiming a 50% improvement is more likely to result in publication than using a realistic baseline and claiming a 0-5% improvement. Unfortunately, they are probably right.

I fear this contradicts your assertion that published results are not garbage. If researchers are claiming their systems are strong based on the fact that they beat weak baselines, then their claims are rubbish and their scholarship shoddy. If, on top of that, their work is difficult to reproduce, or replicate (by the distinction in the paper) then the published results are beyond garbage- they are meaningles garbage, garbage that fails to prove even its weak and insignificant claim to beat a weak baseline.

Basically, this is exactly the kind of shoddy scholarship that Backus accused social scientists of, in his infamous speech about "Cargo-Cult Science". In that speech, he points out that basing new research on previous weak research can only lead in the proliferation of weak results (my interpretation of his words, from memory).

This seems very much to be the case in machine learning research today. This article only provides further evidence that there is something rotten to the core in machine learning research.

>> Reviewers should push back against weak baselines, but be more accepting of results that make modest claims.

Agreed about the second part, however, the paper in the OP says that the way things are, it is very difficult to know which baselines are strong. My interpretation: because they're all trying to beat weak baselines that became established by beating earlier weak baselines. It's weak baselines all the way down.