Well, that's a bit like asking "what's with the obsessive emphasis on programming computers, after all they're just objects that compute?"

Yes, neural networks are objects that compute; there's even this "universal approximator" theorem that says a basic, albeit sufficiently large, neural network can approximate any arbitrary function (from a broad class of functions) to arbitrary precision. However, the theorem says nothing about whether you'll ever actually _find_ the neural network that corresponds to that function. This is what training is for, it allows us to find (the parameters of) the NN that we want to do some computation.

In other words, training is how we program NNs, but in general it can be really hard to arrive at the "program" you're looking for.

> Well, that's a bit like asking "what's with the obsessive emphasis on programming computers, after all they're just objects that compute?"

Indeed! Most of the time computers are computing, not being programmed. Yet most of the time, neural networks are being trained instead of computing.

That was exactly my point.

Your point is that training is computationally intensive, but your question is "why do people obsess over training"? Sounds like you've answered your own question, then. It's currently hard to train networks, so people "obsess" over improving the methods (see, for example, the article we're commenting on) so that it doesn't have to take as long.

But also note that what you say is not necessarily true that the NN's spend most of their time training. Maybe you've got to spend a week on a huge GPU cluster training some autonomous-driving algorithm, but then it runs in "compute" mode for hours a day in tens of thousands of cars.

There’s a universal approximation theorem for polynomials too but that don’t necessarily mean much

The one thing I do not like about neural networks is that they completely abstract the programmer from the problem they are solving. After the neural network is extensively trained, it’s very difficult for even the creator to look at the resulting data structure and make sense of how the problem is actually being solved. I think this is one major reason why neural networks successes have been so confined to specific problems; it’s nearly impossible to predictably modify and improve on a duly functioning and trained multi hidden layer network without starting from scratch and completely retraining it.

What does this mean? I can shape a network some other way?

Yes, you can think very hard and set up the weights by hand, or by an automatic "compiler" that transforms your desired algorithm into a deep net.

Good practitioners do both. Whatever knowledge about the weights you can predetermine, you set those as the initial values of your weights. Then you improve on that by training. It really depends on the problem and size of your network how many weights you'll have a good estimate for but yes, you should try to estimate good weights yourself first.

This practice is well known but here's a concrete source from Andrej Karpathy:

https://karpathy.github.io/2019/04/25/recipe/

"init well. Initialize the final layer weights correctly. E.g. if you are regressing some values that have a mean of 50 then initialize the final bias to 50. If you have an imbalanced dataset of a ratio 1:10 of positives:negatives, set the bias on your logits such that your network predicts probability of 0.1 at initialization. Setting these correctly will speed up convergence and eliminate “hockey stick” loss curves where in the first few iteration your network is basically just learning the bias."

Good luck manually writing a program that achieves state of the art performance on any typical machine learning task... we'll see which "training" method takes longer.