News for hackers. Stuff that matters.

A comment on Understanding the generalization of ‘lottery tickets’ in neural networks. Parent comment

Re:Understanding the generalization of ‘lottery tickets’ in ... by longtom on Tuesday November 26, 2019 @08:09PM (#21644413)
Maybe the additional parameters give the entire network more "leeway" to find such subnetwork structures, i.e. ease gradient descent by smoothing the loss landscape?
[ Parent | on HN ]
Re:Understanding the generalization of ‘lottery tickets’ in ... by Akababa on Tuesday November 26, 2019 @11:43PM (#21645308)
That's intuitive but doesn't support the result of the lucky subnetwork (once found and re-initialized) training faster and outperforming the original.
[ Parent | on HN ]
Re:Understanding the generalization of ‘lottery tickets’ in ... by longtom on Wednesday November 27, 2019 @02:14AM (#21645937)
This does not seem to be a contradiction. Once you are in the right region of solution space training is expected to be faster and easier. Re-initialization could have a regularizing effect, explaining the better performance.
[ Parent | on HN ]
Re:Understanding the generalization of ‘lottery tickets’ in ... by Akababa on Wednesday November 27, 2019 @07:51AM (#21647405)
They re-use the same initialization, so it appears that the initial weights are inherently coupled with the nonzero structure.
[ Parent | on HN ]