Re:Are deep neural networks dramatically overfitted? (2019)
by samcodes on Monday April 05, 2021 @09:31AM
(#26698679)
this is a good explanation of intrinsic dimension
https://eng.uber.com/intrinsic-dimension/
Re:Are deep neural networks dramatically overfitted? (2019)
by qPM9l3XJrF on Wednesday April 07, 2021 @12:11AM
(#26720516)
Thanks. The implication seems to be that if you restrict the # of parameters so it's equal to intrinsic dimension, learning isn't possible, but it is possible with this random projections method. Wonder why. It seems like with both methods, the number of possibilities being explored is the same, but the higher-parameter model space is richer with solutions for some reason.
Re:Are deep neural networks dramatically overfitted? (2019)
by belgian_guy on Monday April 05, 2021 @09:57AM
(#26699027)
The intrinsic dimension paper indeed doesn't really show that big networks also search in small subspaces (and neither does it claim to), but this has already been shown in related papers like https://arxiv.org/abs/1812.04754
Re:Are deep neural networks dramatically overfitted? (2019)
by rocqua on Monday April 05, 2021 @04:45AM
(#26696667)
I think it is related to the lottery ticket idea. Essentially, you try to find the lowest d such that you can find a 'winning ticket' network of size d.
>By searching through the value of d=1,2,…,D, the corresponding d when the solution emerges is defined as the intrinsic dimension.
Is the idea to increase the "intrinsic dimension" until the network is able to learn? I'm not sure I buy the part of the argument where because the network is able to learn when the "intrinsic dimension" is kept low, it therefore follows that the "intrinsic dimension" is also low even when we aren't forcing it to be low. It seems a bit like saying "because we can obtain 90% accuracy on MNIST with a 10-parameter model, it therefore follows that a 1000-parameter model for MNIST only has 10 'intrinsic' parameters". Seems like a dubious / handwavey argument to me.