For instance, I suspect a large fraction of claimed "better than baseline" results in papers are actually a result of well-tuned proposed model and not-very-well-tuned baseline. Papers usually say something along the lines of "we tune these hyperaparameters with crossvalidation", but you can be very thorough or not very thorough in this process and achieve very different results. Or if the author is (intentionally or unintentionally) devious, they could adjust the hyperparameter search ranges in a way that doesn't cover the optimal region, or is way too broad.
All of these problems combined are the reason that I pay much less attention to numbers in the tables of papers. Papers are, unfortunately, more of a source of cool ideas and inspiration for stuff to put into your toolbox and maybe try yourself in your next project.
A year or two may not sound like much but for deep learning but you might as well be comparing Heron's engine[1] to a modern combustible engine. The most cited attention paper, Bahdanau et al. 2014, was only two and a half years ago - and their results weren't all that strong initially anyway. There are so many minor auxiliary improvements / newly found domain expertise that can add up to a substantially improved result (initializations, regularization techniques, hyper parameter choices that work well for that given task / dataset, ...) even for almost the exact same model.
As an example of this, I implemented a Keras SNLI model[2] as a teaching aid for friends only to find my simplest baseline outperformed a number of fairly recent papers. This is not necessarily an issue at all with those papers - many of the ideas and techniques they propose have gone on to be used in other tasks and models - but it does indicate how these additional best practices / more time on the baseline can really accumulate over only months. Keras is a brilliant example of that as it's "best practices included" - when you use an LSTM it'll default to strong weight initializations etc.
I'm vaguely hoping that in the future there can be workshops at conferences titled "Pushing the baselines" where we take some time to see what new techniques can apply to old baselines and/or tune baselines with the explicit goal of getting them to regain their edge. There's at least one at ICML on reproducibility[3] which I'm looking forward to :)
[1]: https://en.wikipedia.org/wiki/Aeolipile
[2]: https://github.com/Smerity/keras_snli
[3]: https://sites.google.com/view/icml-reproducibility-workshop/...
Recently I had to implement gradient calculations by hand recently (writing custom CUDA code) and had a pretty terrible time. Mixing the complications of CUDA code with my iffy manual differentiations and floating point silliness can drive you a little bonkers. I ended up implementing a slow automatic differentiated version and compared resulting outputs and gradients to help work through my bugs.
Here's hoping that Tensorflow's XLA and other JIT style CUDA compilers/optimizers will make much of this obsolete in the near future.
For those not familiar, the overhead for calling a CUDA kernel can be insanely high, especially when you're just doing an elementwise operation such as an add. Given your neural network likely has many many of these, wrapping many of these into one small piece of custom CUDA can result in substantial speed increases. Unfortunately there's not really any automatic way of doing that yet. We're stuck in the days of either writing manual assembly or being fine with suboptimal compiled C.
[1]: https://www.tensorflow.org/versions/r0.11/api_docs/python/te...
[2]: https://github.com/pytorch/pytorch/blob/master/torch/autogra...
Right now we hand write all of our own gradients as well.
The overhead can come from a ton of different places. This is why we wrote workspaces: http://deeplearning4j.org/workspaces
Allocation reduction and op grouping are only a few things you can do.
it is a computer algorithm, so by definition it is trivial to reproduce results, you just run the program again.
> I ran into this recently by accident when writing a simple RL example. With two weight matrices to learn, the first weight matrix was given correct gradients, the last weight matrix was only supplied with partial information. Surprise, still works, and I only discovered the bug _after_ submitting to OpenAI's Gym with quite reasonable results.
so you want to say that you coded a bug, but you don't have a method of testing whether you have a bug. So you didn't code a bug. And if you didn't code a bug, you can't reproduce a bug.
So yes, reproducing a bug is difficult when you have no means of determining whether or not you have a bug...
...maybe you should look into choosing a means of determining whether or not you have a bug.
Not really. Deep learning is still quite a lot of dark voodoo where random initialization and data shuffling can matter significantly. People also adapt hyperparameters manually during training, stop early with no clear metric, and don't share their code for preprocessing the data or even the exact architecture of the network.
It's certainly better than in other fields, but it's not trivial.
If I have a function f and an element of the domain x, then the value y = f(x) does not change. This is not dark voodoo.
I ran into this recently by accident when writing a simple RL example. With two weight matrices to learn, the first weight matrix was given correct gradients, the last weight matrix was only supplied with partial information. Surprise, still works, and I only discovered the bug _after_ submitting to OpenAI's Gym with quite reasonable results. I've seen similar issues in the past such as accidentally leaving a part of the network frozen (i.e. it was randomly initialized and never changed) yet the model still happily went along with it.
This is good and bad. Bad in that it makes errors difficult to catch. Good in that, if you had a reason for freezing part of the network (maybe transfer learning etc) your model will learn to happily use it, even if that "information" is more akin to noise.
Regarding reproducibility, most papers I've gone to reproduce take far longer than expected and usually involve deducing / requesting additional information from the lead authors. Even minor issues, such as how the loss is calculated (loss / (batch * timestep) vs loss / batch) can confuse substantially and given they seem "insignificant" and that there are space constraints in papers, they are rarely written down.
Worst I have seen recently was a state of the art published result where the paper was accepted to a conference yet they didn't include a single hyperparameter for their best performing model - and no code. There is near zero ability to reproduce that given the authors spent a small nuclear reactor worth of compute performing grid search to get the optimal hyperparameters.
tldr There are reproducibility issues all the way up the stack, from gradient descent working against you to minor omissions in the papers to full fledged omissions that are still accepted by the community.