Very nice to see actual theoretical progress being made among the hundreds of “we applied this weird network configuration and it works, we have some intuition but no real understanding”.

The Generalization Gap (test/train) seems to me to be quite unsatisfying as a definition, and then an appeal that theory and practice are different doesn't make me feel that anything is explained.

Being able to create new regularisation techniques is impressive though... looks like something that will need a read!

I haven't read the paper yet, but when people say things like "We don't know why neural networks work", what they mean is we don't understand why they generalise so we'll, by most prior theory they are quite over parameterized, so while they should be able to fit the data - and it has been shown that typical architectures can fit random data - previous theory said the penalty for this is that it will not generalize well to new data from the same distribution.

So generalization bounds on NNs are actually the key thing that people want from theory.

I think the reasoning is that optimizing the training performance is "easy", whereas optimizing the test performance is "hard". If you can guarantee that test performance will be close to training performance, then optimizing the test performance becomes "easy".

An aside: How do you read research papers and how long does it take you to read one paper? Are you able to recall a large portion of the paper's content?

I use a customized version of this: http://blizzard.cs.uwaterloo.ca/keshav/home/Papers/data/07/p...

The most important thing for me is to not read it from the beginning to the end (which is hard for me). Abstract->Conclusions, scan headlines. Methods is the most curious section for me. Depending on the paper and what I am working on I read the methods section last or first (even before the abstract). If it's more of a "oh that seems neat" paper I skip the methods section and mostly extract the idea. The book "How to read a book" is also a good source of ideas.

I don't think there's a one size fits all approach. I also find the various papers on writing literature reviews very helpful (for gathering an overview of a topic). Just checked my Zotero and these are the ones I have tagged:

"Using grounded theory as a method for rigorously reviewing literature"

"On being ‘systematic’in literature reviews in IS"

"A hermeneutic approach for conducting literature reviews and literature searches"

"Systematic literature reviews in software engineering–a systematic literature review"

"Writing narrative literature reviews."

Feel free to go as meta as you want ;)

Proof-heavy papers can take days or weeks to understand. It takes multiple reads, working through the math on scrap paper, and frequently discussing proofs with colleagues.

My process is generally to read through the paper a couple of times and then start tackling the proofs (if there are any). In general, I find it best to start with a very high level understanding and slowly descend into a greater detailed understanding.

> how long does it take you to read one paper?

It really depends. A deep learning paper just presenting a new architecture can be read in a few hours, meanwhile for some heavy papers on the mathematical side can take days, weeks to fully digest.

In the end it doesn't matter, you can be as slow as you want :)

read the abstract, read the conclusion, try to work out what the authors are trying to say.. hunt for the pay off in the paper, see if I can understand that, look for the results and check to see if I can understand that what they claim to show supports the pay off then start reading it. The actual reading can take up to six weeks for a big journal paper.

True.

Skim, index, find context (e.g. Citations to the work if old enough). Some papers take a few years to really understand, which is why mental indexing is important: when you run into a problem solved by the paper, the paper becomes useful and understandable at the same time.

Agree about the years to understand - the important thing is to read it enough so that when you come across something you remember the approach and can option your investment in it.

> how long does it take you to read one paper?

It depends on to what extent the paper is self-contained.

By applying Lemma 4.9 of Lee 2014 to Corollary 6.2.11 of Simpson 1954, but substituting Tao 1977's simplified integral expression in the denominator, the result is obvious.

Why are they using MNIST to demonstrate generalization? Isn't it basically a toy problem at this point?

It is a problem that is well-known and easy to solve. Their point is to show that they offer a boost in terms of training speed.

The empirical results aren't really the "point" of this paper.