"Context drift as data distribution changes" sounds a hell of a lot like real life to me.
Normalized = hedonic treadmill on long view
At the micro scale, data that overflows the normalization is stored in emotional state, creating an orthogonal source of truth that makes up for the lack of full connected learning.
The fact that I argued why I found it bogus based on well established principles, and I get shitted on by people who by all means have provided nothing to this conversation and except suppressing criticism or throwing ad-hominems should tell all about the quality of discourse.
Dismissing criticism, not by arguments, but by the mere name of the person does a disservice to everyone.
If the research can't stand on its own, independent of the author, then it is not good research.
If you can point to _fundamental_ criticism of my arguments, and not fallacies or attacks, I'd be more than happy to discuss them.
With that kind of tonal promise, especially considering the source you are dismissing outright is important in their field, you have to show, not just tell.
If you just left that No out, and gave room for the chance that you are wrong, people wouldn't downvote, they'd upvote. People like to hear smart arguments. No one wants to hear outward dismissal. Especially of known experts.
I could name many other people who have actually been more influential in the field.
Child commenter expressed that the reply may have been sincere and not sarcastic, in which case, my sincerest apologies :(
The claims made are not that deep for researchers.
Since gradients don't flow from B to A in (B.f.A)(x), A is trained independently of B, meaning that the training distribution of B changes without B influencing it, i.e. context drift. B doesn't know the difference, and B doesn't influence it.
For all intents and purposes, you can compute all the outputs as training of A happens, and then feed them into B and B will still compute the same outputs and derivatives as it did before.
To deal with context drift, Hinton proposes normalizing the data, so the distribution does not change significantly.
Whatever he proposed is not "backprop-free" either. It still involves backprop, but the number of layers gradients flow through is 1, the layer itself.
The argument that you can still train through non-differentiable operations is not particularly convincing either; the reparameterization trick shows that is trivial to pass gradients through non differentiable operations if we are smart about it.
Given non differentiable operator Z: R^N -> R^N; let A, B, C be R^N -> R^N linear layer, B(Z(C(x)) * A(C(x))) allows gradients to flow through B and A all the way to C. The output of Z is for all intents and purposes a Hadamard product with (A . C)(x) that is runtime constructed and might as well be part of the input.
You can even run Z(C(x)) through a neural network and learn how to transform that and still provide useful and informative gradients back to C(x) via (A . C)
By looking at it through that perspective, the issues with the approach become evident, and are fundamental in my opinion.
No.
Hinton "discovered" stacking ensembles and gave it a new name, fancy analogies to biological brains and then made it worse.
The gist of this is that you can select a computational unit, be it a linear layer, or a collection of layers, compute the derivative of the output with respect to the parameters, and update them.
Each computational unit is independent, meaning that you don't calculate gradients going outside of it.
This is the same as training a bunch of networks, computing predictions, and then using another layer to combine the predictions. This is called stacking, and the networks are called an "ensemble". You can do this multiple times and have N levels of meta estimators.
Instead of fitting the ensemble and then the meta estimator, Hinton proposes training both simultaneously but without allowing gradients to flow through.
That is stupid because if you don't allow gradients to flow through, you will see a context drift as the data distribution changes. Hinton observed this context drift, to deal with that, he proposed normalizing the data.
On one extreme, you can use individual linear units as the models, and on the other extreme, you can combine all units into a single neural network and treat that as a module.
So no, this does not open any new design, it's an old idea, worsened, and wrapped in fancy words and post-facto reasoning.
If you are curious how a linear layer is an ensemble, observe that each vector is its own linear estimator, making the linear mapping an ensemble of estimators.