> the lottery hypothesis

Isn’t that another way of saying the optimization algorithm used in finding the network‘s weights (gradient descent) can not find the global optimum? I mean this is nothing new, the curse of dimension prevents any numeric optimizer to completely minimize any complicated error function and it’s been known for decades. AFAIK there is no algorithm that can find the global minimum of any function. And this is what currently limits neural network models: They could be much simpler and less resource hungry if we had better optimizers.

I think you're right, but the issue might be local minima which a better optimiser wouldn't help with much. A reason a larger network might work better is that there are fewer local minima in a higher dimension too.

> there are fewer local minima in a higher dimension

Is it actually proven, or another hypothesis? What is the reason behind this?

Just reasoning about this from first principles, but intuitively, the more dimensions you have, the more likely that you are to find a gradient in some dimension. In an N-dimensional space, a local minimum needs to be a minimum in all N dimensions, right? Otherwise the algorithm will keep exploring down the gradient. (Not an expert on this stuff.) The more dimensions there are, the more likely it seems to be that a gradient exists down to some greater minimum from any given point.

In practice, you don't want the global optimum because you can't put all possible inputs in the training data and need your system to "generalize" instead. Global optimum would mean overfitting.

Regularisation should not be done with the optimiser but with the loss function and the architecture.

Regularization only helps you so much.

Maybe it should not be done but the large neutral networks this decade absolutely rely on this. A network at the global minimum of any of the (regularized) loss functions that are used these days would be waaay overfitted.

The entire reason SGD works is because the stochastic nature of updates on minibatches is an implicit regularizer. This one perspective built the foundations for all of modern machine learning.

I completely agree that the most effective regularization is inductive bias in the architecture. But bang for buck, given all the memory/compute savings it accomplishes, SGD is the exemplar of implicit regularization techniques.

Can someone explain this? Isn't it possible for the global optimum to be also be the right generalisation optimum?

it's possible, but unlikely. The issue is your training examples are essentially a noisy representation of the general function you are trying to get it to learn. Generally any representation that fits too well will be incorporating the noise and that will distort the general function (in the case of NN it'll generally mean memorising the input data). Most function-fitting approaches are vulnerable to this.

Hm. I see. But, ultimately, overfitting is a consequence of too many parameters absorbing the noise. Perhaps one could fit smaller models and add artificial noise.

The global optimum would be taken in reference to the training data (because that's all you have to set the weights). Unless the training data represents all real world data perfectly, fully optimizing for it will pessimize the model in relation to some set of real world data.

In addition to that the hypothesis asserts that a local minimum is likely not good enough. This is different from a few years ago when most thought that the solution space was full of local minima so parameter initialization wouldn't matter that much. But that is perhaps because the threshold for acceptable performance is higher so luck is more important.

> once we have trained a large overparameterized neural network we can often compress it to a smaller one with very little loss. So why can't we learn the smaller one directly?

I feel something similar goes on in us humans. Interesting to think about.

There is indeed an analogous process in the brain.

"The number of synapses in the brain reaches its peak around ages 2-3, with about 15,000 synapses per neuron. As adolescents, the brain undergoes synaptic pruning. In adulthood, the brain stabilizes at around 7,500 synapses per neuron, roughly half the peak in early childhood.

This figure can vary based on individual experiences and learning." -- written by GPT-4o

Confirmed by e.g. https://extension.umaine.edu/publications/4356e/

Also perhaps why the evolved pattern of death is important: a subnetwork is selected in a brain, which is suited to a specific geological, physical, biological and cognitive environment that the brain is navigating. But when the environment shifts beneath the organisim (as culture does and the living world in general does), then the subnetwork is no longer the correct one, and needs to be reinitialized.

Or in other words, even in an information theoretic sense, it's true: you can't teach a old dogs new tricks. You need a new dog.

From an evolutionary perspective, wouldn't we expect to have developed a way to "reset" parts of our brain then?

That is what kids are for.

Haha, exactly. "We did".

This reminds me of a hacker news comment that blew my mind - basically "I" am really my genetic code, and this particular body "I" am in is just another computer that the code has been moved to, because the old one is scheduled to be decommissioned. So I am really just the latest instance of a program that has been running continuously since the first DNA/RNA molecules started to replicate.

If you're interested in such things, then start layering on epigenetics. The "I" is a product not just of genes, but of your environment as you developed. I was just reading about bees' "royal jelly" recently, and how genetically identical larvae can become a queen or a worker based on their exposure to it.

So the program is not just the zeroes and ones, so to speak, but also more nebulous real-time activity, passed on through time. Like a wave on the ocean.

And my children are not only my offspring, but everyone I come into memetic contact with.

I like you.

Oh! This reminds me of some lines from a 1970s essay called "The Origin of Death", which was on HN 5 months ago:

https://news.ycombinator.com/item?id=38505856 https://www.elijahwald.com/origin.html

> The strange thing about all this is that we already have immortality, but in the wrong place. We have it in the germ plasm; we want it in the soma, in the body. We have fallen in love with the body. That’s that thing that looks back at us from the mirror. That’s the repository of that lovely identity that you keep chasing all your life. And as for that potentially immortal germ plasm, where that is one hundred years, one thousand years, ten thousand years hence, hardly interests us.

> I used to think that way, too, but I don’t any longer. You see, every creature alive on the earth today represents an unbroken line of life that stretches back to the first primitive organism to appear on this planet; and that is about three billion years. That really is immortality. For if that line of life had ever broken, how could we be here? All that time, our germ plasm has been living the life of those single-celled creatures, the protozoa, reproducing by simple division, and occasionally going through the process of syngamy -- the fusion of two cells to form one—in the act of sexual reproduction. All that time, ^^that germ plasm has been making bodies and casting them off in the act of dying. If the germ plasm wants to swim in the ocean, it makes itself a fish; if the germ plasm wants to fly in the air, it makes itself a bird. If it wants to go to Harvard, it makes itself a man.^^ #weirding The strangest thing of all is that the germ plasm that we carry around within us has done all those things. There was a time, hundreds of millions of years ago, when it was making fish. Then at a later time it was making amphibia, things like salamanders; and then at a still later time it was making reptiles. Then it made mammals, and now it’s making men. If we only have the restraint and good sense to leave it alone, heaven knows what it will make in ages to come.

> I, too, used to think that we had our immortality in the wrong place, but I don’t think so any longer. I think it’s in the right place. I think that is the only kind of immortality worth having -- and we have it.

YES. great response

Neuroplasticity is a thing though, with plenty of cases of brains recovering from pretty significant damage. They also do evolve and adjust over time to gradual changes in environment. Lots of elderly people are keeping up with cultural and technological change.

Can't say the same about neural networks (yet?).

Indeed, you have to over-engineer something before converge to a leaner solution to a problem

And I think this is because the ideal complex system is one where all the subsystems and parts combine to produce adequate reliability.

Exponentiation means it is more efficient to start by far exceeding the required reliability and then optimizing the most expensive subsystems/parts. It is less efficient and far more frustrating if multiple things have to be improved to meet requirements.

This is really insightful.

Yes. My naive intuition about this is you need the extra parameters precisely to do the learning because learning a thing is more complicated than doing the thing once you have learned how. There are lots of natural examples that fit this intuition eg in my mind "junk" DNA is needed because the evolutionary mechanism is learning the sequences which work in a similar way. You don't need all that extra DNA once you have it working but once you have it working there's little selection pressure to clean up/optimise the DNA sequence so the junk stays.

Thanks, very clear explanation!

This may be a silly question but - so rather than train a big network and hope a subnetwork wins the lottery - why not just train a smaller network with multiple runs with different starting weights?

Largely for the same reason people don't "just win" the lottery by buying every possible ticket (any more).

The larger network contains exponentially more subnetworks. 10x the size contains far more than 10x subnetworks (although it'd also take more than 10x as long to train).

Ah ok - so you are saying the explosion of subnetworks is higher than the explosion of training time - leading to a win.

In this case a positive benefit of combinatorial complexity.

> So why can't we learn the smaller one directly?

The lottery hypothesis intuitively makes sense, but as an outsider I find this concept for evaluating learning methods really interesting - To hand craft a tiny optimal networks for simple yet computationally irreducible problems like GoL as a way to benchmark learning algorithms. Or is it more than that? for a sufficiently small network maybe there aren't that many combinations of "correct solutions", so perhaps the way the network emerges internally could really be interrogated by comparison.

Isn’t this basically the idea behind dropout technique?

No,the idea behind dropout is to reduce an over-reliance on specific outputs thereby, in theory and typically in practice, making the network learn more reliable representations reducing the chance of overfitting.