News for hackers. Stuff that matters.
Re:Llama3 implemented from scratch by delusional on Sunday May 19, 2024 @04:22PM (#40409496)
The innovation is the amount of resources people are willing to spend right now. From looking at the research code it's clear that the whole field is basically doing a (somewhat) guided search in the entire space of possible layer permutations.

There seems to be no rhyme or reason, no scientific insight, no analysis. They just try a million different permutations, and whatever scores the highest on the benchmarks gets published.

Re:Llama3 implemented from scratch by moffkalast on Sunday May 19, 2024 @04:34PM (#40409554)
Well it took evolution 4 billion years of testing out random permutations that resulted in a pretty good local maximum, so there is hope for us yet.
Re:Llama3 implemented from scratch by WanderPanda on Sunday May 19, 2024 @06:47PM (#40410316)
„I‘m a pretty good local maximum“ that is what any local maximum would tell you if asked how it likes itself
Re:Llama3 implemented from scratch by moffkalast on Monday May 20, 2024 @04:03AM (#40413331)
"The brain is the most important part of the body", the brain said.
Re:Llama3 implemented from scratch by psychoslave on Monday May 20, 2024 @09:02AM (#40415015)
Note that not all brains are so severely damaged with this illusion. Most of them actually get pretty clearly that they are next to useless without its organic, social and environmental companions.
[dead]
Re:Llama3 implemented from scratch by stefs on Sunday May 19, 2024 @05:10PM (#40409771)
Or maybe a watch made by a blind guy instead of a bridge?
Re:Llama3 implemented from scratch by moffkalast on Sunday May 19, 2024 @05:24PM (#40409851)
Would that make it more of a hear? Since obviously the guy can't watch.
Re:Llama3 implemented from scratch by ben_w on Sunday May 19, 2024 @06:33PM (#40410254)
I can't see the flagged comment, but I'm fairly sure @stefs is making a reference to this: https://en.wikipedia.org/wiki/The_Blind_Watchmaker
Re:Llama3 implemented from scratch by stefs on Monday May 20, 2024 @02:43PM (#40418664)
yes. the flagged comment wanted to sell moffkalast a bridge, w/o further explanation. i interpreted it as him saying that the human brain isn't governed by evolution and calling moffkalast naive.
Re:Llama3 implemented from scratch by killerstorm on Monday May 20, 2024 @07:00AM (#40414178)
There's definitely scientific insight and analysis.

E.g. "In-context Learning and Induction Heads" is an excellent paper.

Another paper ("ROME") https://arxiv.org/abs/2202.05262 formulates hypothesis over how these models store information, and provide experimental evidence.

The thing is, a 3-layer MLP is basically an associative memory + a bit of compute. People understand that if you stack enough of them you can compute or memorize pretty much anything.

Attention provides information routing. Again, that is pretty well-understood.

The rest is basically finding an optimal trade-off. These trade-off are based on insights based on experimental data.

So this architecture is not so much accidental as it is general.

Specific representations used by MLPs are poorly understood, but there's definitely a progress on understanding them from first principles by building specialized models.

Re:Llama3 implemented from scratch by inciampati on Monday May 20, 2024 @07:39AM (#40414392)
One 3-layer (1 hidden layer) neural network can already approximate anything. You don't even need to stack them.