News for hackers. Stuff that matters.
Re:Modern language models refute Chomsky’s approach to language by asdff on Tuesday June 13, 2023 @02:57PM (#36315097)
Imagine being told all you need to do to learn Spanish, is to read a 300,000 word Spanish dictionary end to end so that you can probalistically come up with 1000 conversational phrases. Anyone who has learned a language can tell you it just doesn't work like that. You don't work by accumulating a massive dataset and training on it. No one can hold such a massive dataset of anything in their head at once.
Re:Modern language models refute Chomsky’s approach to language by kristopolous on Tuesday June 13, 2023 @03:00PM (#36315152)
Going from zero to one is different than one to two ("second language acquisition")

Zero to one is closer to mimicry and immersion. There's a long Wikipedia article on the field of study https://en.m.wikipedia.org/wiki/Language_acquisition

Furthermore, humans probably aren't static learners and likely have more beneficial times of certain study than others. There's a theory in that too https://en.m.wikipedia.org/wiki/Critical_period_hypothesis

Saying there's a "digital brain" is more of a framework since the term "brain" looks like it's a moving target

In another comment I referred to these systems as like comparing hydraulic pumps to human biceps, cars to horses, etc.

We can use the same units of measure, give them the same tasks, but saying they're the same thing only works in the world of poetry

Re:Modern language models refute Chomsky’s approach to language by jackmott42 on Tuesday June 13, 2023 @03:02PM (#36315179)
The LLM doesn't hold the whole dataset in its head either.
Re:Modern language models refute Chomsky’s approach to language by aezart on Tuesday June 13, 2023 @03:24PM (#36315560)
Not for a lack of trying.

If you give it a small enough training set or a big enough neural network, it will directly memorize the whole thing. You have to intentionally make its brain too small to do that in order to force it to find patterns in the data instead.