Re:Do Deep Nets Really Need to be Deep? [pdf]
by sgt101 on Sunday October 05, 2014 @03:40AM
(#8411706)
Just checking what you are saying in your last paragraph: do you mean to say that shallow NN's don't overfit?
[ Parent
| on HN ]
Re:Do Deep Nets Really Need to be Deep? [pdf]
by wodenokoto on Monday October 06, 2014 @01:01AM
(#8414353)
No, shallow NN definitely overfits in normal circumstances. However, in this case they are building a shallow NN that is not learning a normal function, but is being trained to emulate a deep network of equivalent amount of parameters.
Parents claim is that in that case, we don't have to worry about over fitting, since we want every result of target network to be as closely emulated as possible in the shallow network.
Re:Do Deep Nets Really Need to be Deep? [pdf]
by Houshalter on Sunday October 05, 2014 @01:55PM
(#8412885)
If I understand correctly, they generated artificial data from the NN they were mimicking.