Re:Proving the Lottery Ticket Hypothesis: Pruning is All You...
by anonymousDan on Saturday March 07, 2020 @05:06PM
(#22514516)
As a potentially naive thought experiment, if you just generated in advance a number of random networks of similar size to the pruned lottery ticket, and then trained them all in parallel, would you eventually find a lottery ticket? If so how many would you have to train to find a lottery ticket with high probability? Why is training one big network and then pruning any better than training lots of different smaller network? Assume in all of the above that you have a rough idea of how big the pruned network will be be.
[ Parent
| on HN ]
Re:Proving the Lottery Ticket Hypothesis: Pruning is All You...
by aerodude on Sunday March 08, 2020 @12:38AM
(#22516520)
The number of subgraphs increases exponentially with the number of additional layers (and neurons). If you started off with a network the size of the final pruned network, you would have a dramatically lower chance of finding a winning ticket compared to the oversized network you start with.