News for hackers. Stuff that matters.
Re:Do Deep Nets Really Need to be Deep? [pdf] by ivan_ah on Saturday October 04, 2014 @05:14PM (#8410680)
Very interesting and well written paper. They show a deep-neural network (which people thought had more explaining-power-per-parameter) can be "emulated" by a shallow neural network with the same number of parameters.

What Ba and Caruana have shown is that deep neural networks---as architecture---are not more powerful. The previous general opinion (at least this is what I thought) was that deep architectures of the form:

  predictions
  layer 3  features of features of features
  layer 2  features of features 
  layer 1  features 
  data layer  
would be more efficient (i.e. more explaining power per parameter). Nope. Apparently, a shallow neural network of the form

  predictions
  layer 1  features   (a lot more of them)
  data layer  
with an equal number of units in layer 1 as the total number of units in deep network, can obtain the same accuracy. Thus, deep neural networks and shallow neural networks have roughly equivalent explaining-power-per-parameter.

A separate result shows deep convolutional neural networks can be simulated by shallow NNs with a simulation overhead of ~10x.

Another cool observation they make about "installing" the parameters of ML models. We'll use the following analogy to illustrate the point:

  ML model               <==>  hardware 
  ML model parameters    <==>  software 
Running program X involves (1) buying/renting a server, (2) compiling X for your architecture, and (3) running X. Of course there is always the alternate step (2') of installing pre-compiled binaries.

Similarly, we can think of using a ML model as a three step procedure (1) ML.init() (malloc for Model.parameters), (2) Model.train() (fit parameters to training data), (3) Model.predict(new_datum).

The authors point out the existence of a possibility of distributing models parameters as binaries. They demonstrate an alternate step (2') in which they train a ''FancyModel'', then set the parameters of the simple model so it "simulates" the FancyModel: ''Model.parameters = simulate(FancyModel)''. In other words, the model parameters of a ML model need not be learned through the model itself.

What I find fascinating is it seems deep architectures don't have any more representational power, but they do have more learning power. Something very interesting is going on...

Re:Do Deep Nets Really Need to be Deep? [pdf] by amalcon on Saturday October 04, 2014 @09:39PM (#8411184)
The novel part is not that the shallow network can compute the same functions (that is interesting, but that particular result is decades old), but the notion that the shallow network can learn the same functions efficiently.

What we already knew: For any given neural network of N layers, there exists a 3-layer neural network computing the same function.

What we didn't know: There are (at least reasonably often) ways of training shallow networks that get similar efficiency to the ways of training deep networks given the same functions.

Re:Do Deep Nets Really Need to be Deep? [pdf] by Houshalter on Saturday October 04, 2014 @11:09PM (#8411340)
No, they require training a deep neural network first. Then training the shallow net to mimic the deep one.

The claim is that the shallow net can have the same number of parameters as the deep net. It's always been known that a large enough shallow NN can theoretically approximate any function. But that it can do so with few parameters is very surprising.

My best guess is that the majority of parameters in deep NNs are unused or redundant. I.e they have a low weight, or don't influence the output very much, or another neuron computes mostly the same function. Many types of regularization like dropOut and weight decay heavily encourage this.

Whereas the shallow NN is forced the maximize the usefulness of every single parameter to get the same result. It doesn't have to worry about overfitting. I am not sure if the paper accounted for this though, it's been awhile since I read it.

Re:Do Deep Nets Really Need to be Deep? [pdf] by sgt101 on Sunday October 05, 2014 @03:40AM (#8411706)
Just checking what you are saying in your last paragraph: do you mean to say that shallow NN's don't overfit?
Re:Do Deep Nets Really Need to be Deep? [pdf] by wodenokoto on Monday October 06, 2014 @01:01AM (#8414353)
No, shallow NN definitely overfits in normal circumstances. However, in this case they are building a shallow NN that is not learning a normal function, but is being trained to emulate a deep network of equivalent amount of parameters.

Parents claim is that in that case, we don't have to worry about over fitting, since we want every result of target network to be as closely emulated as possible in the shallow network.

Re:Do Deep Nets Really Need to be Deep? [pdf] by Houshalter on Sunday October 05, 2014 @01:55PM (#8412885)
If I understand correctly, they generated artificial data from the NN they were mimicking.
Re:Do Deep Nets Really Need to be Deep? [pdf] by tiger10guy on Saturday October 04, 2014 @09:58PM (#8411216)
Their method for training a shallow network requires that one first train a deep network.
Re:Do Deep Nets Really Need to be Deep? [pdf] by dougabug on Sunday October 05, 2014 @06:31AM (#8411934)
It seems reasonable that when you decompose the representation of the target function you are trying to learn into more layers, each of the layers can be "more gently" modulated to emulate the target function as observed through the training data. Differential techniques might therefore more readily find good approximations to the target function. Once the structure has been captured by finely slicing it into more layers, this structure can be compressed down into a shallower representation.