News for hackers. Stuff that matters.
Re:When to assume neural networks can solve a problem by eugenhotaj on Monday March 30, 2020 @10:21AM (#22728218)
I disagree, PAC bounds are usually way too loose to be useful in practice when it comes to NNs. I think part of the reason is that the way we measure model complexity for NNs does not correspond very well with their ability to generalize, as we've seen with recent papers like deep double descent [1].

Anecdotally, I was of the main authors of the AdaNet framework [2] where we used (approximate) Rademacher complexity (closely related to PAC) to bound the complexity of our learned ensemble models. We never got good results using the complexity measure and would basically turn it off for any practical problem we solved.

[1]: https://openai.com/blog/deep-double-descent

[2]: https://github.com/tensorflow/adanet

There are no comments on this comment yet.