An additional frustration point I always have is when network architectures are not even fully specified.
Try reading the MTCNN fave detection paper. How, exactly, is the input image pyramid calculated? By what mechanism, exactly, can the network cascade produce multiple detections (i.e. can it only produce one detection per each input scale? If more than one, how?). In the Inception paper dealing with factorized convolutions, just google around to see the deep, deep confusion over the exact mechanics by which the two-stage, smaller convolutions ends up saving operations ovrr a one-stage larger convolution. The highest upvoted answers on Stack Overflow, reddit, quora are often wrong.
And these examples are from reasonably interesting mainstream papers that deserve some fanfare. Just imagine how much worse it is for extremely incremental engineering papers trying to milk the hype by claiming state of the art performance.
Still though, at the end of the day, I’d rather that more papers are published and negative / incremental results are not penalized, because the alternative file drawer bias would be much worse for science overall.
(the evil?) Elsevier has a similar opinion => https://www.elsevier.com/connect/scientists-we-want-your-neg...
But this criticism falls a little flat to me. For instance,
> “Nevertheless the central point of a scientific paper is a relatively concisely expressible idea of some nontrivial universality (and predictive power) or some nontrivial observation about the nature of reality”
That’s an insanely high bar for published work. I also read lots of research papers, and I think only a handful per year would meet these requirements. Yet many others are extremely valuable to show negative or partial results, results with small effect sizes, and other things.
We absolutely should not disparage someone for publishing results of a failed or ineffectual approach. Because otherwise we’ll just make things like file drawer bias and p-hacking far worse, and create an even worse cultural expectation that to make a career in science, you must constantly publish positive results with big, sexy implications — which is what leads to the whole disastrous hype-driven state of affairs, like in deep learning right now, in the first place, and ludicrous science journalism, funding battles fought over demoware and vaporware, academics fleeing into corporate sponsorship like yesterday’s article about Facebook, etc.