Would it make sense for them to add a gallery of good solutions for each problem, or would they all basically be your second example network (no time to play and see for myself right now)?
It's probably worth pointing out that this is true for ANNs, but there were (and are) other "shallow" classifiers that can handle swiss roll problem without manual parameter encoding. SVMs, for example.
If N levels off, then the network has grasped the concept of a spiral and can generalize to arbitrary size.
If N doesn't level off, then the network isn't really learning the general case.
So, I ported their swiss roll dataset to python and threw together a shallow network trainer with theano:
https://gist.github.com/notmatthancock/68d52af2e8cde7fbff1c9...
Then, I trained a shallow network with 36 hidden units (your deep net has 6 units and 6 layers):
http://i.imgur.com/I0pXaTK.png
edit: I forgot to mention that the shallow network above takes only the two coordinates (x1 and x2) as input features.
It feels like neurons in the first layer are weaker, because all they can do is a linear separation. Given deep networks, I was wondering if adding neurons to the first layer was better than adding them to the last one, and empirically, it feels like it is quite worse. I wonder if there is a theorem around that.
Correct, but keep in mind that their method appears to use batch descent while mine does not. Batch descent is often converges more quickly. There are other differences between my net and the GP's I can spot as well (e.g., the activation function, the learning rate, and regularization).
Also keep in mind that I threw this together over breakfast, and did not spend much time tweaking parameters :)
Also how do you know to choose a ReLu instead of a Tanh activation?
6 layers is the maximum that this demonstration allows, and they kept j small-ish to show that you don't need that many to have good results.
It may just be that 'batched cumulative learning' (I don't know if there is already a term for this) gets a better fit than just learning from a smaller set of data.
Edit: Did a quick test; regenerating about every 50 and 100 iterations, and conversion does seem faster (at least, when a clear spiral is formed). https://imgur.com/a/OPjXb
In a normal situation, you obtain a list of input / output (say, images as input, a digit as output, for learning handwritten digits). You separate it between training data (which actually improves the net) and testing data (to detect overfitting), and you don't get more data than that.
Here, you can generate more data for free, as we have the function we want to approximate. Having more data will often result in a better result and faster convergence.
I tried the swiss roll with a shallow network on the demo (and the results are not excellent, but it matches)
Before deep learning people would manually design all these extra features sin(x_1), x_1^2, etc. because they thought it was necessary to fit this swiss roll dataset. So they would use a shallow network with all these features like this: http://imgur.com/H1cvt8d
Then the deep learning guys realized that you don't have to engineer all these extra features, you can just use basic features x_1, x_2 and let the network learn more complicated transformations in subsequent layers. So they would use a deep network with only x_1, x_2 as inputs: http://imgur.com/XBRjROP
Both these approaches work here (loss < 0.01). The difference is that for the first one you have to manually choose the extra features sin(x_1), x_1^2, ... for each problem. And the more complicated the problem the harder it is to design good features. People in the computer vision community spent years and years trying to design good features for e.g. object recognition. But finally some people realized that deep networks could learn these features themselves. And that's the main idea in deep learning.