It’s more that without backpropagation, you no longer need to store your forward activations across many layers to compute the backwards pass which usually is dependent on a forward pass. When a network is hundreds of layers, and batches are very large, the forwards and backwards accumulations add up in terms of memory required.

Communication across GPUs doesn’t solve this but instead allows to have either many models running in parallel on different GPUs to increase Barch size or to share many layers across GPUs to increase model size. Quick communication is critical to maintain training speeds that aren’t astronomical

> will this algorithm uncover a new design space for NNs?

No.

Hinton "discovered" stacking ensembles and gave it a new name, fancy analogies to biological brains and then made it worse.

The gist of this is that you can select a computational unit, be it a linear layer, or a collection of layers, compute the derivative of the output with respect to the parameters, and update them.

Each computational unit is independent, meaning that you don't calculate gradients going outside of it.

This is the same as training a bunch of networks, computing predictions, and then using another layer to combine the predictions. This is called stacking, and the networks are called an "ensemble". You can do this multiple times and have N levels of meta estimators.

Instead of fitting the ensemble and then the meta estimator, Hinton proposes training both simultaneously but without allowing gradients to flow through.

That is stupid because if you don't allow gradients to flow through, you will see a context drift as the data distribution changes. Hinton observed this context drift, to deal with that, he proposed normalizing the data.

On one extreme, you can use individual linear units as the models, and on the other extreme, you can combine all units into a single neural network and treat that as a module.

So no, this does not open any new design, it's an old idea, worsened, and wrapped in fancy words and post-facto reasoning.

If you are curious how a linear layer is an ensemble, observe that each vector is its own linear estimator, making the linear mapping an ensemble of estimators.

Thats a lot less cheaty, biologically speaking, than full backprop. This Hinton guy sounds like he knows what he's talking about.

"Context drift as data distribution changes" sounds a hell of a lot like real life to me.

Normalized = hedonic treadmill on long view

At the micro scale, data that overflows the normalization is stored in emotional state, creating an orthogonal source of truth that makes up for the lack of full connected learning.

You're accusing one of the foundations of modern AI with either being a fraud, or incompetent. At best that seems short sighted, no?

You're obviously new to Hacker News :-D

Unfortunately, I've been here for a decade in one form or another. Every now and then someone writes something so pompous that I just can't help myself but post. Back to lurking now. Cheers!

If we can't publicly scrutinize people who have great sway in the industry, what does that say about us as a research community?

The fact that I argued why I found it bogus based on well established principles, and I get shitted on by people who by all means have provided nothing to this conversation and except suppressing criticism or throwing ad-hominems should tell all about the quality of discourse.

Dismissing criticism, not by arguments, but by the mere name of the person does a disservice to everyone.

If the research can't stand on its own, independent of the author, then it is not good research.

We are all looking forward to your research paper that disproves his claims. Or you know, any proof.

I argued for it and all I got was downvoted without criticism of the substance of my arguments, only ad-hominems and fallacies.

If you can point to _fundamental_ criticism of my arguments, and not fallacies or attacks, I'd be more than happy to discuss them.

The hard dismissal with 'No' is likely why you got down voted. I am not able to do that.

With that kind of tonal promise, especially considering the source you are dismissing outright is important in their field, you have to show, not just tell.

If you just left that No out, and gave room for the chance that you are wrong, people wouldn't downvote, they'd upvote. People like to hear smart arguments. No one wants to hear outward dismissal. Especially of known experts.

I am neither the first, nor the last who believe that the Laureates have not done their due diligence properly with respect to citing sources.

I could name many other people who have actually been more influential in the field.

It's really quite impressive how you know everything about ML, considering your own professed basic level of expertise.

https://news.ycombinator.com/item?id=33583593

Edit:

Child commenter expressed that the reply may have been sincere and not sarcastic, in which case, my sincerest apologies :(

Maybe you know something but I don't, but I believe the comment you are replying to was a compliment and not a sarcastic dig.

This person has made a similar remark and purposefully looked into my post history - which doesn't make any claims about my knowledge or skills - to make a sarcastic remark, see "everything".

A lot can change in 60 days.

They do have a master's degree according to the post.

The claims made are not that deep for researchers.

The use of the word masters is now considered not cool according to Stanford... :-)

It's not like I have been doing many things outside reading papers and books over the past few years... The post that person used as an ad-hominem even says so.

Yeah, no. Reading the paper I don't really see anything but a superficial resemblance to stacking. Hinton was active back when Wolpert introduced stacking and I'm fairly sure he is aware of it. If anything it much more closely resembles his own prior work in Boltzmann machines, unsurprisingly (and which he cites), or even his prior work on capsules. I don't know if this will really pan out into anything that different or useful for the field, but it's unfair and inaccurate to dismiss it as derivative of stacking.

A single linear layer is for all intents and purposes equivalent to running an ensemble of linear estimators. By disallowing gradients to flow between two layers A, B, computing (B . f . A)(x) with f being a non linearity, the second layer is an ensemble of linear estimators of the outputs of the first, and for all intents and purposes, the output of (f.A)(x) is just preprocessing for B.

Since gradients don't flow from B to A in (B.f.A)(x), A is trained independently of B, meaning that the training distribution of B changes without B influencing it, i.e. context drift. B doesn't know the difference, and B doesn't influence it.

For all intents and purposes, you can compute all the outputs as training of A happens, and then feed them into B and B will still compute the same outputs and derivatives as it did before.

To deal with context drift, Hinton proposes normalizing the data, so the distribution does not change significantly.

Whatever he proposed is not "backprop-free" either. It still involves backprop, but the number of layers gradients flow through is 1, the layer itself.

The argument that you can still train through non-differentiable operations is not particularly convincing either; the reparameterization trick shows that is trivial to pass gradients through non differentiable operations if we are smart about it.

Given non differentiable operator Z: R^N -> R^N; let A, B, C be R^N -> R^N linear layer, B(Z(C(x)) * A(C(x))) allows gradients to flow through B and A all the way to C. The output of Z is for all intents and purposes a Hadamard product with (A . C)(x) that is runtime constructed and might as well be part of the input.

You can even run Z(C(x)) through a neural network and learn how to transform that and still provide useful and informative gradients back to C(x) via (A . C)

I'm not sure what the main point is here. The paper is definitely sketchy on details, and the main idea is definitely simple enough to resemble a lot of other work. I wouldn't be surprised if someone (maybe a certain Swiss researcher) comes out and says, actually, this is the same as this other paper from the early 90s. If you squint hard enough a lot of ideas (especially simple ones) can be seen as being the same as other, older ideas. I'm not too interested in splitting those hairs, really. I'm more curious on whether this eventually leads to something that sets it apart from the SOTA in some interesting way.

My claim is that this work is simply worse ensembles wrapped in a biologically inspired claims, and that arguments made in support of it by the author compared to other approaches are simply not sound.

By looking at it through that perspective, the issues with the approach become evident, and are fundamental in my opinion.