"Every time I fire a linguist, the performance of the speech recognizer goes up". - Frederick Jelinek
"It would be interesting to know how different a model would be if it operated on eg dependency trees instead of the linear list of tokens." - indeed, this is obviously interesting, so people have tried that a lot for many models, but IMHO it's probably now almost decade since the consensus is that in general end-to-end training (once we became able to do it) work better than adding explicit stages in between, e.g. for any random task I would expect that doing text->syntax tree->outcome is going to get worse results than text->outcome, because even if the task really needs syntax, the syntax representation that a stack of large transformer layers learns implicitly tends to be a better representation of the natural language than any human linguist devised formal grammar, which inevitably has to mangle the actual languge to fit into neat human-analyzable 'boxes'/classes/types in which it doesn't really fit and all the fuzzy edge cases stick out. Once you remove the constraint that the grammar must be simplified enough for a human to be able to rationally analyze and understand, and (perhaps even more importantly?) abandoning the need to prematurely disambiguate the utterance to a single syntax tree instead of accepting that parts of it are ambiguous (but not equally likely), processing works better.
It's just one more reminder of the bitter lesson (http://incompleteideas.net/IncIdeas/BitterLesson.html) which we don't want to accept.
So there seems to be such emergent mechanisms in the model that have arisen because of the end-to-end training, which we don't exactly understand yet.
First off, we know that overall the concept of language is humans is an emergent phenomenon. It developed from natural selection from simple components so there's validity that the same thing can occur in an LLM where some overarching emergent structure develops from simple primitives.
At the same time we do know that a sort of universal grammar exists among humans. Our language capacity is biased in a certain way and that it is unlikely for it to learn languages of a very extreme and divergent grammar from the universal one discovered by Noam chomsky. That means our brain is unlikely to be as universally simple as an LLM.
I think the key here is that the human mind has explicit linguistic tools but the these tools are still emergent in nature.
Has it changed since then? Have they found a significant number of human civilizations that use divergent grammars?
Harris’s operator grammar was based on set theory however Chomsky was enamored with formal logic and ran in that direction. Also, Chomsky became famous while Harris didn’t.
Operator grammar is self-discoverable and he published a full description of English grammar in the 1980’s using this theory and an extension in the 1990’s which generalized to other languages.
It is not fully deterministic (final word selection and ordering is probabilistic), but it is much more convincing to me and in line with how we understand brains to work.
Over the last 20 years Chomsky has begun to fall out of favor because it’s just so complicated and requires external structures, etc.
Any theory that explains all the facts is valid, there can be more than one competing theory. Fame of the author absolutely matters and so does elegance of the theory.
Chomsky has held on so long because it did pretty well and he became very famous, but over time it has needed larger and larger patches to cover its flaws, so other explanations are gaining traction once again.
No the elegance of the theory does not matter. The truth of the theory does. I'm saying does the universal grammar you describe, is it actually universal? Amongst all the languages in the world, do they all share this other grammar you describe? If not then the theory is incorrect. It doesn't matter how elegant the theory is.
I brought this up specifically because your reasoning wasn't "scientific" it was more along the lines of logical elegance.
>Chomsky has held on so long because it did pretty well and he became very famous, but over time it has needed larger and larger patches to cover its flaws, so other explanations are gaining traction once again.
See this makes sense and is basically the answer to the question I am asking. So you're basically saying that there were flaws? As in he had to make up new rules constantly because the science was contradicting his theory. Am I correct in this characterization?
I’m not an expert, but if I recall I think the big flaw was insistence on determinism.
And to my knowledge operator grammar is universal to the degree there isn’t a counter example in the dozen of languages he explored and hundreds he had others help him with. But to my knowledge the only language he fully mapped was English.
Edit: take a look at the article on Wikipedia.
https://arxiv.org/abs/1905.05950
I think it's a mistake to discount the psychology of learning. It's like saying that "calories-in, calories-out" is all there is to weight-loss. Strictly true, but not helpful for 90% of people.
Probability and observation are all that is required to understand a language.
I don’t think this is entirely fair. Generative grammars have been produced for a huge variety of non-European languages, even non-Indo-European languages, and can account for tremendous diversity in linguistic rules. Even languages without fixed word orders or highly synthetic languages can be represented.
Linguistics isn’t focused on the problem of outputting reasonable-sounding text responses. Instead, it seeks to transparently explain how language works and is structured, something that GPT does not do.
>grammar as we know it was devised for the Latin language and linguists spend most of the time attempting to fit other languages into neat boxes that the Latin grammar wasn't designed for....Chomsky tried to . All solve this problem with Universal Grammar.
Well first of all, every language has some form of grammar. If there were no rules that made meaning depend on word order, there would be nothing different between what the cat ate and what ate the cat. To claim Grammar is some Latin/Western system being unduly applied is absurd.
You are correct UG was invented as a to explain something about English. Howevert was not an attempt to solve "this problem ("this problem" being conforming to expected systems of Grmmar.) The problem he was interested in solving was about the ability to learn language rapidly despite not having many negative examples of how not to talk. Pay attention here researchers, the lesson maybe extends to you too soon.
UG also isn't a specific enough thing you can check against in a directory. Its a theory, namely a theory that if you study language features you'll eventually discver some things are invaiant. Using it directly and immediately based on nothing but Chomsky's conjecture...all I can focus on passing out at screen goodnightn