News for hackers. Stuff that matters.
Re:We Found an Neuron in GPT-2 by Baeocystin on Thursday February 16, 2023 @08:57PM (#34829675)
Approaches such as you describe have been the dominant method for decades. That we finally 'cracked' natural language generation with tools that literally encode nothing about grammar ahead of time is one hell of a lesson, early days as it is in the learning of it.
Re:We Found an Neuron in GPT-2 by letmevoteplease on Thursday February 16, 2023 @09:04PM (#34829751)
Reminds me of Stephen Krashen's input hypothesis of second-language acquisition. Krashen argues that consciously studying grammar is more or less useless, and only massive exposure to the language results in acquisition.[1] This is true in my experience.

[1] https://en.wikipedia.org/wiki/Input_hypothesis

Re:We Found an Neuron in GPT-2 by nmfisher on Thursday February 16, 2023 @09:46PM (#34830060)
I'm designing a Chinese learning app and I'd mostly agree with this. My working hypothesis though? Most adult learners don't have the time/patience for massive/lengthy exposure, so grammar lessons are a shortcut to "feeling" like they're making progress.

I think it's a mistake to discount the psychology of learning. It's like saying that "calories-in, calories-out" is all there is to weight-loss. Strictly true, but not helpful for 90% of people.

Re:We Found an Neuron in GPT-2 by charcircuit on Thursday February 16, 2023 @11:46PM (#34830896)
Grammar lessons are a shortcut. So are books and other corpuses of knowledge. People have spent time documenting patterns that exist and its useful to learn from them instead having to brute force everything yourself.
Re:We Found an Neuron in GPT-2 by naasking on Friday February 17, 2023 @08:19AM (#34834299)
This is assuming that the brain regions that learn rules overlap with the brain regions that develops fluency in a language. I think Krashen's hypothesis is that this is largely not the case. You can "fake" some degree of competence by learning the rules and using that brain region, but you're slow and not fluent until you expose the other region to enough real-world data.
Re:We Found an Neuron in GPT-2 by sitkack on Friday February 17, 2023 @01:49AM (#34831628)
Ref, A Bitter Lesson
Re:We Found an Neuron in GPT-2 by solarmist on Friday February 17, 2023 @03:24PM (#34840042)
Harris, Chomsky’s advisor, published his operator grammar over the decades and one of the key features was that it is completely self-discoverable.

Probability and observation are all that is required to understand a language.