NaiveEmpiricist

It looks like another difference with RAGs is that instead of reranking the doc vectors and taking the top 5 or so, this paper mixes all the docs (weighted by relevance) into a fixed number of "memory tokens".

It looks like this helps avoid a problem I've had with RAG - when a fact isn't local to a single patch of text.

I think you just explained the situation, even if you didn't realize it.

LLMs process millions of times more text data and use millions of times more compute power to learn a language, mostly because we know how to do that technologically.

Children use much less text data, but also use a lot of other kinds of data, including interacting with other humans, to learn language. We have no idea how that works. And, on top of that, people can interact with pets of various species in the same way as children, but those pets never learn language.

So, something strange is going on that children can do, but that LLMs and puppies can't. Probably, part of it is that human children are much better at learning by copying what adults are doing, and part of it is that humans evolved to learn and use language. The details of what's going on there is what linguistics is about.

For decades, Chomsky was the most-cited author who still had a pulse. And a lot of that was for his political writing - maybe 2/3rds? But he still has a lot of cites for his intellectual work, in several different fields.

Linguistics, obviously.

Philosophers love him, because he always gives them an argument, and that's all they really care about.

There probably wouldn't even be cognitive science without Chomsky. And psychology is still responding to Chomsky's nuking of behaviorism in the 1950's.

So, there's a lot there, but admittedly most of it is for politics.

Actually, there have been a number of rounds of paradigm shift (or attempted paradigm shift) in linguistics over the past 70 years. The unique thing about linguistics is that they've been compressed into the lifetime of the same central figure.

Also, I'll mention that there is a lot of empirical knowledge about natural language syntax in generative grammar, even if the details aren't quite right about their theoretical underpinnings. And a lot of them were "low-hanging fruit" waiting for someone to point out how to find them and characterize them. That Chomsky did that is the best argument for his intellectual value - permitting all that "normal science" to happen.

Science doesn't guarantee technological applications, whether or not that science is correct.

Finally, just saying: use of a word like "Chomskyism" makes you sound like one of those creationists who call evolution "Darwinism".

Indeed. Who'd have thought that mere statistics could do that? Hat's off to them - they made the darn thing work.

There's pretty interesting (I think) research in how effectively it's possible to scale down LLMs. If you're interested, take a look at the BabyLM conferences:

https://babylm.github.io/

(I have no connection to them.) The work there demonstrates that gap between what we can do now with human-relevant amounts of training text, and what remains to be discovered. Quite a bit, it looks like.

Also, AFAIK, insects rely on their (flat) flapping wings to create lift by creating vortices.

"Nothing" may be a little too strong -- physics is physics, after all -- but (small) birds and airplanes have Reynolds numbers about 1000x different and different underlying material science.

Airplanes are engineered (with a very small number of exceptions) to be stable during flight, whereas bird flight is inherently unstable, requiring a lot of specialized cognition so birds can fly.

I would think a lot of cognitive ornithologists (who I really hope exist!) would be interested in finding out about this.

At this point, I'm at the complete end of my knowledge of bird flight!

I haven't seen the evidence for "raped kids", but it's certainly true that Chomsky advised and benefitted from Epstein long after any sensible person would have realized that Epstein JUST MIGHT be a bad person.

Unfortunately, Chomsky's always had a problem pulling the ripcord in similar situations. As one of his colleagues once explained, "When most people step in dogshit on the sidewalk, they scrape it off their shoe and keep walking. Chomsky gets stubborn and just keeps stepping in it again and again."

Not that this is relevant to his intellectual contributions, unless it crosses your personal line where involvement becomes complicity.

> The only thing LLMs show is that it is possible to learn language de-novo.

I agree, but would add, "But not under conditions that humans learn language."

Norvig (and Chomsky) are talking at cross purposes.

Norvig confuses engineering for science. If an engineer wants something that works (and guarantees that it will work) then you use the most effective tool available to achieve that goal. No problem there. But there's absolutely no reason to think that tells you anything scientific about the phenomenon you're modeling. For example, aeronautical engineering says very little about how birds fly. And - currently relevant - LLMs learn to use and understand language, but they use millions of times more training data, and millions of times more computing power, than a child does. So an LLM may be a great technology, but it's probably not a good model for how humans learn and use language.

Chomsky OTOH doesn't see any use for statistics in science, at least not in linguistics. Fortunately for the field of linguistics, Chomsky's influence in linguistics is waning and linguists are beginning to use statistical tools to do science.

"Unproven" is in the eye of the beholder, but to me it certainly looks like the sudden take-off in arithmetic ability (to name one example) at approx 10^22 training FLOPs across all LLMs is emergent. (ref. https://hai.stanford.edu/news/examining-emergent-abilities-l...)

I'd be interested to see where this goes, but it seems to rely on LLMs being mostly or entirely compositional (i.e. ultimately, they can be taken apart to be understood). But if LLMs have notable emergent properties - which seems likely - then those parts of the LLM will be effectively off-limits to this sort of interpretation.

And emergence doesn't need to involve much of the LLM's weights:

> In preliminary audits of Qwen3-32B, we estimate that roughly 90% of neuron poles have one main job, often with small related offshoots; another ~7% have two related jobs; and only ~2-4% are not yet understood

If those ~2-4% are the central hubs of emergent cognition, then understanding 96%-98% of the weights may be less valuable than the author thinks.

Butlerian Jihad?