How do neural networks learn?

Re:How do neural networks learn? by HarHarVeryFunny on Tuesday March 19, 2024 @01:28PM (#39758169)

You're talking about something a bit different - dependence on how the NN is initialized, etc. When people talk about "emergent properties" of LLMs, this is not what they are talking about - they are talking about specific capabilities that the net has that were not anticipated. For example, LLMs can translate between different languages, but were not trained to do this - this would be considered as an emergent property.

Nobody is saying this is magic - it's just something that is (with our current level of knowledge) impossible to predict will happen. If you scale a model up, and/or give it more training data, then it'll usually get better at what it could already do, but it may also develop some new (emergent) capabilities that no-one had anticipated.

[ Parent | on HN ]

Re:How do neural networks learn? by wredue on Thursday March 21, 2024 @11:02AM (#39779368)

Finding unexpected connections is something we’ve known LLMs are good at for ages. Connecting things you didn’t even know are connected is like “selling LLM business to business 101”. It’s the first line of a sales pitch dude.

And that’s still beside the point that the properties that emerge can greatly differ just by changing the ordering of your training.

Again, we see this on NNs training to play games. The strategies that emerge are completely unexpected, and when you train a NN multiple times, often differ greatly, or slightly.

[ Parent | on HN ]