> And this is exactly a problem of collapse of knowledge: you can't ask an LM to code you a problem it hasn't seen several dozen of times during training. You just cannot, you will not get a satisfying answer.

I think that's more a function of the coding ability of the model you are using, and obviously they are going to do much better on languages better represented in the training set.

What the paper is referring to as "knowledge collapse" isn't inability to do something, but rather getting a mid response since by definition that's going to dominate any normal (pardon the pun!) distribution.

Here's what he said:

"While large language models are trained on vast amounts of diverse data, they naturally generate output towards the 'center' of the distribution."

But, you can generally prompt around this sort of thing by asking the model to adopt a preferred context such as "you're an expert xxx", providing of course that is did see such expert-tagged material in the training set.

> But, you can generally prompt around this sort of thing by asking the model to adopt a preferred context such as "you're an expert xxx", providing of course that is did see such expert-tagged material in the training set.

So, I need LM to be an expert in SAT solving and an expert Haskell programmer. This means I need LM to be trained on the SAT solving and on vast amount of Haskell code.

I think it is already easy to see that LM should be trained in an intersection of quite fringe (yes, I am watching that TV series right now) things.

I also think that it is beyond reason to ask for this.

Please get back to us when you try whatever you’re trying to do with a good model (gpt4, opus, or ultra).

I recommend refraining from making (and especially from posting) any conclusions until then.

> a good model

Strongly reminds me of "no true scotsman."

If you have an access to a good model, try that simple problem of blocked clause decomposition and please get back to us.

I do have access to good models, but you are the one making far reaching statements about LLMs.

Please preface your future posts on the topic with “I’ve never tried any decent models, so I don’t know what I’m talking about”.

I will include such disclaimer after you prove me wrong as you have means to do that.

Sounds good. What prompt do you want me to try?

"Please implement blocked clause decomposition in Haskell"

I, actually, asked my friend to ask whatever LLM he paid for to perform this experiment. I think he uses latest or previous GPT. At least he is paying for it which I cannot do and won't do in foreseeable future.

Just like I said above, not only LLM failed to properly implement the algorithm, it failed to provide even remotely correct types for clauses and blocked clauses.

What happened next amused me like nothing else related to AI.

The exchange that followed incorrect initial attempt, GPT clearly tried to persuade my friend to implement the correct version for it, constantly providing him with partially (and stupidly) incorrect code.

This is exactly what MathCraft [1] does.

[1] https://en.wikipedia.org/wiki/Cyc#MathCraft

And this behavior is clealy counterproductive. This is not what one might expect from anything that is advertized as a good assisting measure. I can use ghci, vim and actual paper with much greater effect.

The LLM under test was GPT 4 turbo.

>This means I need LM to be trained on the SAT solving and on vast amount of Haskell code.

Essentially why the current batch of top-tier LLMs are trained on a pile of text amounting to “the internet”, containing every haskell q&a on stackoverflow, every sat implementation on github, and every explaination of those terms on wikipedia. It might seem unreasonable without the sense of scale and expense of building these things; but yes, it has those input data in its training set.

Blocked clause decomposition is implemented in, I think, two SAT preprocessors in SAT solvers, both are C++. There are about three to five different articles on the topic.

For LLM to implement BCD in Haskell it needs to be able to translate between Haskell and C++ and it is well beyond trivial.

> obviously they are going to do much better on languages better represented in the training set.

Which, again, indicates that the current crop of LLM’s are just fuzzy-recollection/fuzzy-interpolation machines, and have none of the actual thought/reasoning capabilities some people are claiming they have.

If that were the case, it would be impossible to describe rules to them and have them follow those rules.

I think Chomsky's idea that LLM's are a form of automated plagiarism is the best description if we think of plagiarism in a neutral sense and not a highly negative sense.

If we think in terms of automated plagiarism, no one can complain that the model doesn't produce original ideas.

Automated plagiarism could have many uses but it is obviously stupid to expect an original idea from plagiarism. We wouldn't complain the model can't reproduce things it hasn't seen before if we think in terms of plagiarism.

Chomsky has been wrong for decades. He doesn't want to believe that LLMs are really learning language because that conflicts with his assertion that it takes something much more specialized to do so.

Basically Chompsky has been proved wrong by LLMs, but rather than believing what is in front of his eyes he is saying "that can't be right, because it'd mean I am wrong".

“Chomsky is wrong because I say he is, and because he doesn’t like the LLM I like”

Mmmhhmmm yes very convincing argument there. I’d be willing to bet that Chomsky’s arguments don’t rely on such spurious arguments. Automated plagiarism reads like a fantastic description of current LLM’s to me.

Do you disagree that LLMs appear to have learnt language? Whether they plagiarize or not is irrelevant as long as they are not quoting verbatim. If they are coming up with their own sequence of words, making sense, and remaining grammatical, then how can you argue that they have not learnt language?

Since they do at least (whatever other shortcomings they may have) appear to have learnt language, just by "listening to" a bunch of examples, and since they are based on a transformer, not some specialized Chompskian "language organ", then the thesis that a "language organ" is needed is disproved.

Even assuming LLMs have learnt language (whatever that means) it is completely irrelevant to what Chomsky is studying which is how the human language ability works. As an analogy training a neural network to successfully predict the weather doesn't falsify physics-based models of the weather.

Sure, but for our cortex not to be able to learn language without becoming specialized for it, while a transformer can, implies that a transformer is a more powerful prediction engine than our cortex, which seems unlikely.

If you look at the cortex as a whole, what seems most striking is how uniform it is - same basic architecture of six layers of neurons with a specific pattern of inter-layer connectivity. It seems nature has come up with a universal prediction architecture. Maybe Wernicke's areas is fine-tuned for language, but to characterize it as a specialized "language organ" seems a bit of a stretch. Let's note too that language is only a million years old, while the cortex itself is 100's of millions, yet has this mostly uniform architecture that evidentially works just as well for vision as hearing, etc, etc.

So, sure, the ability of the ridiculously simple transformer architecture to learn language (and many other prediction tasks you throw at it), doesn't PROVE that the brain didn't have to evolve a highly specialized way to do it, but it certainly seems highly suggestive of it.

Since we now have an existence proof that a very simple architecture, not specialized for language, can learn language, it seems the onus is now on Chompsky to put some meat on the bones of his claim (without evidence) that the general cortical architecture is incapable of this without a high degree of specialization.

> Sure, but for our cortex not to be able to learn language without becoming specialized for it, while a transformer can, implies that a transformer is a more powerful prediction engine than our cortex, which seems unlikely.

On the contrary, it's extremely likely that we can engineer something better than random evolution. And we have: calculators are better at adding and subtracting that humans are, bikes are faster than running, etc.

> If you look at the cortex as a whole, what seems most striking is how uniform it is - same basic architecture of six layers of neurons with a specific pattern of inter-layer connectivity. It seems nature has come up with a universal prediction architecture. Maybe Wernicke's areas is fine-tuned for language, but to characterize it as a specialized "language organ" seems a bit of a stretch. Let's note too that language is only a million years old, while the cortex itself is 100's of millions, yet has this mostly uniform architecture that evidentially works just as well for vision as hearing, etc, etc.

Language is ~100,000 years old, not a million. The brain is in fact not uniform: If you damage Broca's area you lose language capability. And to say that all cognitive function is the same general algorithm ignores the obvious fact that the brain performs different functions and doesn't answer the question of how and why this is so. There are lots of cognitive behaviour that is not understood, if you want to explain those behaviors you implicitly have to distinguish them from one another.

> So, sure, the ability of the ridiculously simple transformer architecture to learn language (and many other prediction tasks you throw at it), doesn't PROVE that the brain didn't have to evolve a highly specialized way to do it, but it certainly seems highly suggestive of it.

It doesn't. Moro showed in a series of experiments that humans have difficulty learning non-hierarchical languages and use different parts of the brain to do so, which is highly suggestive that language is specialized.

> Since we now have an existence proof that a very simple architecture, not specialized for language, can learn language, it seems the onus is now on Chompsky to put some meat on the bones of his claim (without evidence) that the general cortical architecture is incapable of this without a high degree of specialization.

He has provided evidence and arguments, some of which I have pointed to above. Maybe you should actually read or listen to him.

> Moro showed in a series of experiments that humans have difficulty learning non-hierarchical languages and use different parts of the brain to do so, which is highly suggestive that language is specialized.

The cortex is built for hierarchical processing, because that's what's needed to model the world we live in. Physical objects are localized and have hierarchical detail, and larger visual scenes are the same. The kind of sequential (temporal) patterns relevant to us are also hierarchical, whether in visual, auditory or other domains.

The type of connectionist architecture needed to recognize hierarchical patterns is a layered one where the receptive field and hierarchical level of abstraction grows as you ascend the layers. In our brain those layers come from different patches of our cortical sheet being connected. This is the reason that neural network architectures like CNNs and transformers also work to recognize hierarchical patterns in visual and temporal domains - because they both also uses these layered architectures, which is all that is needed.

The reason why the function of damaged brain areas can't always be taken over by other areas is largely due to plasticity. Our brain peaks in it's ability to form new synapses in the first few years of life. If you haven't learned language by age of 3, then you will never be able to learn more than a crude type of pidgin language, depsite all your "language areas" being intact. The same would be true of different part of you brain trying to learn language as an adult - the plasticity is no longer there.

If that were the case, then you wouldn't get the transfer learning from models being trained on programming languages, to reasoning applied in a non-programming context, which has been universally observed for all these models. They are learning abstract "thought patterns", not just interpolation.

Learning a programming language (or any complex skill) just by reading a book and looking at examples, without actually practicing it yourself, is basically impossible, even for ourselves, let alone an LLM. It is amazing how well the LLMs can do at programming in cases where they were trained on masses of well-commented examples, but without also being able to learn "on the job" themselves, you need to tamper expectations.

I uploaded the paper to chatgpt and it summarized it, wrote pseudo code and wrote haskell code with very little prompting from me... no idea if it's _good_ code, because I didn't run it and I don't know the domain or haskell, but it did produce code -- and explained that some of it is non trivial to implement and might need more attention and explained why.

But this is _exactly_ the problem being discussed.

Domain expert: “I cannot get it to produce something that even approximately resembles x”

Person: “I gave it the paper and I got maybe x?”

There is a fundamental knowledge gap here, just because you can get it to do something isn’t the goal, or even a good outcome. The illusion that it’s capable of doing something it evidently cannot is a dangerous one, because people who don’t know better, and are being convinced that they do understand or comprehend when they don’t.

Okay so you have no idea how good the output is.

I believe if you pay for Phind Pro it'll give you access to GPT-4 and Claude Opus, which are the two most capable models.

If you are just using the free version of Phind, then it uses some MUCH smaller and less capable model of their own. Even if you use Phind Pro, or Phind Plus, it says their own most capable model is Phind 70B (tiny compared to GPT-4 class models).

You can't just use Phind and take that as indicative of what state-of-the-art models can do, without knowing what model you are actually using. You also can't expect any of these models to do as well with Haskell as they do with Python or Java which they will have seen massively more of during training.

> You also can't expect any of these models to do as well with Haskell as they do with Python or Java which they will have seen massively more of during training.

This is exactly the person’s point.

Did you try this Haskell problem with a paid version and can provide an example that actually disputes the described experience? Why would someone pay for the premium version of a product when they don’t perceive the free version as working?