I haven't read the paper yet, but when people say things like "We don't know why neural networks work", what they mean is we don't understand why they generalise so we'll, by most prior theory they are quite over parameterized, so while they should be able to fit the data - and it has been shown that typical architectures can fit random data - previous theory said the penalty for this is that it will not generalize well to new data from the same distribution.

So generalization bounds on NNs are actually the key thing that people want from theory.

I think the reasoning is that optimizing the training performance is "easy", whereas optimizing the test performance is "hard". If you can guarantee that test performance will be close to training performance, then optimizing the test performance becomes "easy".