News for hackers. Stuff that matters.

A comment on Pruning AI networks without impacting performance. Parent comment

Re:Pruning AI networks without impacting performance by visarga on Wednesday December 13, 2017 @12:14AM (#15912187)
> I wonder if you could also receive similar results by simply turning some connections "off" for a while, only training the rest of the connections. Then turn the missing connections on a gain and, randomly turning a few other connections off, and continue training.

Isn't that almost like Dropout? Dropout does what you say but it changes the dropped set of neurons for each individual example.

[ Parent | on HN ]
Re:Pruning AI networks without impacting performance by vanderZwan on Wednesday December 13, 2017 @05:50AM (#15913436)
https://www.cs.toronto.edu/~hinton/absps/JMLRdropout.pdf

Looks pretty close, yeah.

So this is basically dividing the network into two groups, and give each group a second weight: one for those that are on, zero for those that are off.

Has anyone tried more variants of this? Using random bundling + various levels of amplification of such bundles, so that backpropagation affects various networks differently?

[ Parent | on HN ]
Re:Pruning AI networks without impacting performance by visarga on Friday December 15, 2017 @03:31PM (#15934822)
There is also this: Mixture of experts. A massive neural net from Google, with multiple submodules that are activated depending on input.

https://medium.com/@thoszymkowiak/google-brains-new-super-fa...

[ Parent | on HN ]