Re:Deep Learning Pioneer Geoffrey Hinton Publishes New Deep ...
by hgsgm on Thursday January 12, 2023 @08:39AM
(#34353292)
This command is a lot of words to say "I don't like it" without giving any reason to believe you.
If it's not new or novel, why aren't people using it? If it's bad, what's wrong with it?
Re:Deep Learning Pioneer Geoffrey Hinton Publishes New Deep ...
by LudwigNagasena on Thursday January 12, 2023 @09:00AM
(#34353550)
It performs worse than baсkprop.
Re:Deep Learning Pioneer Geoffrey Hinton Publishes New Deep ...
by qorrect on Thursday January 12, 2023 @09:19AM
(#34353805)
Is it slower and less accurate ? Or just slower.
Re:Deep Learning Pioneer Geoffrey Hinton Publishes New Deep ...
by LudwigNagasena on Thursday January 12, 2023 @10:30AM
(#34354963)
It is far less accurate compared to SOTA models. The paper says it should train faster, but it doesn't provide any metrics; so it's hard to make any "pound for pound" comparisons.
Re:Deep Learning Pioneer Geoffrey Hinton Publishes New Deep ...
by UncleEntity on Thursday January 12, 2023 @10:25AM
(#34354854)
So the initial “hey, this might be a good idea” implementation performs slightly worse than something that has had literally billions of dollars thrown at it?
Re:Deep Learning Pioneer Geoffrey Hinton Publishes New Deep ...
by ekleraki on Thursday January 12, 2023 @10:49AM
(#34355286)
It is slower to the same backprop that we have used for decades now.
No comparisons to AdamW were made.
In fact, this algorithm uses backprop at its core, but propagating through 0 layers.
Re:Deep Learning Pioneer Geoffrey Hinton Publishes New Deep ...
by LudwigNagasena on Thursday January 12, 2023 @10:51AM
(#34355334)
The question was "If it's not new or novel, why aren't people using it?". For example, take a look at this paper: https://arxiv.org/abs/1905.11786. It was published 3 years ago, it also does parallel layer-wise optimization, it even talks about being inspired by biology, though the objective function is different from Hinton's. Why aren't people using it? Because it performs worse. It is that simple. Is it an interesting area to explore? Probably. There are millions of interesting areas to explore. It doesn't mean it is worth using, at least yet.
Re:Deep Learning Pioneer Geoffrey Hinton Publishes New Deep ...
by oneoff786 on Thursday January 12, 2023 @04:28PM
(#34360487)
Very significantly worse.
Re:Deep Learning Pioneer Geoffrey Hinton Publishes New Deep ...
by goethes_kind on Thursday January 12, 2023 @08:55AM
(#34353480)
Since you seem to understand what he is saying, can you explain to me how the per layer objective function looks like?
I don't get what he means by inserting the label into the input and what labels he is using per layer.
Re:Deep Learning Pioneer Geoffrey Hinton Publishes New Deep ...
by LudwigNagasena on Thursday January 12, 2023 @09:12AM
(#34353729)
A model is a function F that minimizes error in y_i = F(X_i) + error. Inserting a label simply means a function F(X_i,y_j). Then you optimize it in some way to separate true labels from false label, e.g. F(X_i, y_j) = (y_i==y_j)-(y_i!=y_j) + error.
Re:Deep Learning Pioneer Geoffrey Hinton Publishes New Deep ...
by bjourne on Thursday January 12, 2023 @11:13AM
(#34355745)
You train the network to detect correlations between the values of the ten first pixels and the rest of the image. Imagine you have a bunch of images of digits. For images with digit three you set the third pixel to white, for images of the digit four, you set the fourth pixel to white, and so on (actually, zero-indexing so fourth and fifth pixel for digit three and four but whatever). The other nine pixels among the first ten you set to black. These are positive samples and training the network with them will make it output a big number when it encounters them. Then you swap the pixels so that the images with the digit three has the fourth pixel set to white and the images with the digit four has the third pixel set to white. These are negative samples and they cause the network to output a small number. Thus, the only difference between positive and negative samples is the location of the white pixel. So for an image you want to classify you run it through the network ten times and each time shifting the location of the white pixel. The location for which the network outputs the biggest number is the predicted class.
Obviously, this method is problematic if you have thousands of labels or if your network is not a classifier.
Re:Deep Learning Pioneer Geoffrey Hinton Publishes New Deep ...
by LudwigNagasena on Thursday January 12, 2023 @08:59AM
(#34353527)
That's the same impression I had. I was afraid I am not getting something or missing a bigger picture. I am glad I am not the only one who feels that way.
As far as I can tell, this is almost the same as stacking multiple layers of ensembles, except worse as each ensemble is trained while previous ensembles are learning. This is causing context drift.
To deal with the context drift, Hinton proposes to normalise the output.
This isn't anything new or novel. Expressing "ThIs LoOkS sImIlAr To HoW cOgNiTiOn WoRkS" to make it sound impressive doesn't make it impressive or good by any stretch of the imagination.
Hinton just took something that existed for a long time, made it worse, gave it a different name and wrapped it in a paper under his name.
With every paper I am more convinced that the Laureates don't deserve the award.
Sorry, this "paper" smells from a mile away, and the fact that it is upvoted as much shows that people will upvote anything if they see a pretty name attached.