Notes on Weight Initialization for Deep Neural Networks

Re:Notes on Weight Initialization for Deep Neural Networks by fromthestart on Monday May 20, 2019 @07:56PM (#19966474)

Actually, there's already a better way to perform initialization, based on the so called lottery ticket hypothesis [1]. I haven't gotten to the article, so I'll just regurgitate the abstract, but basically there frequently are subnetworks which may be exposed by pruning trained networks which perform on par with full sizes neural nets with ≈20% of the parameters and substantially quicker training time. It turns out that with some magic algorithm described in the paper, one can initialize weights to quickly find these "winning tickets" to drastically reduce neural network size and training time.

1. https://arxiv.org/abs/1803.03635

[ Parent | on HN ]

Re:Notes on Weight Initialization for Deep Neural Networks by iXce on Tuesday May 21, 2019 @12:51AM (#19967629)

As far as I understand there is no quick magic algorithm to find them: you train the full architecture as usual the long and hard way, then you identify the right subnetwork and you can retrain faster from the architecture and initialization of just this subnetwork

[ Parent | on HN ]

Re:Notes on Weight Initialization for Deep Neural Networks by L2R on Wednesday May 22, 2019 @01:20PM (#19983284)

Based off of the results, you have to train a larger number of architectures to identify the right subnetwork.

[ Parent | on HN ]

Re:Notes on Weight Initialization for Deep Neural Networks by hnaccy on Monday May 20, 2019 @08:53PM (#19966752)

This paper had trouble getting this to work with lager models I believe.

https://arxiv.org/abs/1902.09574

[ Parent | on HN ]

Re:Notes on Weight Initialization for Deep Neural Networks by p1esk on Tuesday May 21, 2019 @12:06AM (#19967503)

You mean this one https://arxiv.org/abs/1903.01611 ?

[ Parent | on HN ]

Re:Notes on Weight Initialization for Deep Neural Networks by hnaccy on Tuesday May 21, 2019 @02:28AM (#19967975)

No. From my link:

>Additionally, we provide strong counterexamples to two recently proposed theories that models learned through pruning techniques can be trained from scratch to the same test set performance of a model learned with sparsification as part of the optimization process. Our results highlight the need for large-scale benchmarks in sparsification and model compression.

[ Parent | on HN ]

Re:Notes on Weight Initialization for Deep Neural Networks by axiom92 on Monday May 20, 2019 @08:00PM (#19966500)

It sounds very cool. This work also won the best paper award at ICLR 2019.

[ Parent | on HN ]