News for hackers. Stuff that matters.
Re:Natural language instructions induce generalization in ne... by worstspotgain on Tuesday March 19, 2024 @04:44PM (#39760015)
Birds inspired planes. Later, aerodynamics fed back into ornithology. I've been waiting for LLMs to be evaluated as a model of human thought. Complexity and scale have held neuroscience back. It's nowhere close to building a high-level brain model up from biological primitives. Like ornithology, it could use some feedback.

All arguments about AGI aside, a machine was built that writes like a human. Its design is very un-biological in places, so it's tempting to dismiss it. Why not see how deep the rabbit hole goes?

Like all conjectures it just might surprise us. For one, the intuition that language is central to thought predates LLMs, but it's certainly consistent with it.

Re:Natural language instructions induce generalization in ne... by visarga on Tuesday March 19, 2024 @05:39PM (#39760534)
> a machine was built that writes like a human

But the real hero here is not the LLM, but the training set. It took ages to collect all the knowledge, ideas and methods we put in books. It cost a lot of human effort to provide the data. Without the data we would have nothing. Without GPT we could use RWKV, Mamba, S4, etc and still get similar results. It's the data not the model.

> the intuition that language is central to thought predates LLMs

Language carries AI and humans. The same distribution of language can be the software running in our brains and in LLMs. I think humans act like conditional language models with multi modality and actions. We use language to plan and solve our problem, work together and learn (a lot) from others.

Language itself is an evolutionary system and a self replicator. Its speed is much faster than biology. We've been on the language exponential for millennia, but just now hit the critical mass for LLMs to be possible.

It's not so important that GPT-4 is a 2T weights model, what matters is that it was trained on 13T tokens of human experience and it now "writes like a human". Does that mean humans also learn the same skills GPT-4 has learned from its training set mostly by language as well?

Re:Natural language instructions induce generalization in ne... by AlecSchueler on Tuesday March 19, 2024 @06:35PM (#39761026)
> But the real hero here is not the LLM, but the training set.

And in the case of windmills the hero is the wind. But the mill is still a fantastic achievement.

Re:Natural language instructions induce generalization in ne... by riwsky on Tuesday March 19, 2024 @07:48PM (#39761538)
And if you rearrange the letters in “mill”? “I, LLM.”

qed

Re:Natural language instructions induce generalization in ne... by razodactyl on Wednesday March 20, 2024 @01:26AM (#39763090)
Correlation checks out.
Re:Natural language instructions induce generalization in ne... by mewpmewp2 on Tuesday March 19, 2024 @07:25PM (#39761376)
But can mills easily move? No, people can, so people are still viable.
Re:Natural language instructions induce generalization in ne... by coldtea on Tuesday March 19, 2024 @09:07PM (#39761989)
>But can mills easily move?

Yes, that's their whole thing: moving when the air blows.

Re:Natural language instructions induce generalization in ne... by mewpmewp2 on Wednesday March 20, 2024 @05:46AM (#39764457)
But can they relocate?
Re:Natural language instructions induce generalization in ne... by nickpsecurity on Tuesday March 19, 2024 @10:47PM (#39762453)
Indeed, they are mostly parroting that knowledge with some reasoning from combinations of those patterns. That counts for something. It’s not what we do, though. Or not all we do.

Humans can just sit around aimlessly toying with stuff, reading books, etc. They’ll figure out some of these patterns on their own. Whereas, we have to give these things a ton of highly-curated, pre-processed data made by human minds of all kinds. Then, it’s usually 800GB-4TB for the good ones. They’re appear to be not in our league yet as learning machines.

We’ll be able to assess it better as multimodal models come online. We can train them like infants, then children, on books, random observations through cameras, TV, people reading to them, supervised feedback… all the stuff we do with humans. Then see if and how they match up in performance.

Re:Natural language instructions induce generalization in ne... by smokel on Tuesday March 19, 2024 @05:15PM (#39760288)
> Later, aerodynamics fed back into ornithology.

You mean to imply that birds have learned from jet fighter designs?

I fail to understand the point you're making.

Re:Natural language instructions induce generalization in ne... by vaidhy on Tuesday March 19, 2024 @05:26PM (#39760405)
I believe the point is that we understand the birds' flight better applying the principles we learnt designing and flying airplanes. Similarly, we can learn more about human brain by applying things we learn from building the ANN back to the study of humans (anthropology in general, maybe neuroscience and psychology )
Re:Natural language instructions induce generalization in ne... by worstspotgain on Tuesday March 19, 2024 @05:24PM (#39760388)
Concepts from the mechanics of flight (e.g. lift and drag) have helped ornithologists understand birds better. Birds themselves did not learn much.
Re:Natural language instructions induce generalization in ne... by Sardtok on Tuesday March 19, 2024 @05:29PM (#39760436)
He didn't say it fed back into birds, but into the study of birds.
Re:Natural language instructions induce generalization in ne... by throwaway_6142 on Tuesday March 19, 2024 @10:41PM (#39762425)
Knowledge learned from airplane propeller development, as well as from jet plane wing aerodynamics, has had a great impact on the design of wind turbine blades, which are now affecting the evolution of birds.
Re:Natural language instructions induce generalization in ne... by lo_zamoyski on Tuesday March 19, 2024 @05:38PM (#39760527)
Why assume this is sensible? Aerodynamics can help discover general principles that apply to both birds and planes or whatever else. I don't see how this holds for LLMs and brains. The similarity between LLMs and brains is superficial.

Besides, with human beings, we have a host of philosophical problems that undermine the neuroscientific presumption that a mechanistic and closed view of the brain can account for mental activity entirely, like the problem of intentionality.

Re:Natural language instructions induce generalization in ne... by og_kalu on Tuesday March 19, 2024 @05:48PM (#39760633)
>I don't see how this holds for LLMs and brains. The similarity between LLMs and brains is superficial.

It's really not any more superficial than planes and bird flight.