Re:How do neural networks learn?
by hnuser123456 on Tuesday March 19, 2024 @08:34AM
(#39755059)
It's curve fitting at its core. It just so happens that when you fit a function with enough parameters, you can start using much more abstract and higher-level methods to more quickly describe other outcomes and applications of this curve fitting. Really simple algebra just with a lot of variables. It's not a black box at all and it's disingenuous when people call it that. It outputs what it does because it's multiplying A and B and adding C to the input you gave it.
[ Parent
| on HN ]
That's not a useful description, it's the equivalent of explaining the entire software stack as a pile of logic gates.
"It's all just 0s and 1s."
This is a misunderstanding. You're right that you can always find some weights that approximate any reasonable (that is, continuous) mathematical function. The problem is, you have to find those weights in a small amount of computation time, and you need to estimate them from only finitely many samples. In general, ***no neural network architecture can solve this problem efficiently in every single case***. See "No Free Lunch theorem". The fact that current architectures work well in practice, despite the NFL theorem - for seemingly difficult problems - is surprising, and demands an explanation.
Appendix: By "current architecture", I mean Transformers plus Stochastic Gradient Descent.