ChatGPT models semantic relationships in the data. That's what your smart buddy means by "understanding". That is a high dimensional model of the data set which infers abstract semantic relationships / concepts. But he would not claim that those semantic relationships are exactly the same as any human interpretation of the data (which you refer to as the "real" response).
Language models also have limited reasoning abilities. They are capable of misunderstanding as much as understanding.
> Language models also have limited reasoning abilities.
They have no reasoning abilities at all.
Asserted without evidence.
I'm willing to accept that most human speech behavior may well happen in similar ways to LLMs. But most human speech behavior is not reasoning.
Posted by Jason Wei and Denny Zhou, Research Scientists, Google Research, Brain team
https://ai.googleblog.com/2022/05/language-models-perform-re...
At no point does the LLM know that 5+6 = 11, and if asked to solve a problem in which 5+6 was an implicit component of the solution but not explicitly present in the text, it would be completely lost.
> At no point does the LLM know that 5+6 = 11
Does it need to "know" that (by your narrow definition of "know") in order to reason about a word math problem?
> if asked to solve a problem in which 5+6 was an implicit component of the solution but not explicitly present in the text, it would be completely lost
Can you provide an example? What makes you believe it can't be trained to solve those too? That's just a higher abstraction over the language. Add more layers, more training, etc. Many humans cannot solve basic math word puzzles that this artificial neural network can already solve.
If I ask you to multiply two (largeish) numbers together, you will be able to do so, using an algorithm/process that you can apply to the multiplication of any two numbers, whether anyone has ever told you about those numbers before or not.
LLM's cannot do this. Give them a math problem that doesn't exist in their training set and they cannot solve it. This has been demonstrated many times.
They routinely solve math problems (and other reasoning tasks) that don't exist in their training set. Examples were in that paper I linked to. This is one of the incredible emergent properties of LLMs / deep neural networks.
Try it out today on GPT-4. Make up your own math problems and go for it.
RESPONSE: The product of 19192920 and 190101271 is: 3653424693064240
Fail on first try.
It can solve math word puzzles that are not in its training set.
Yes you can find these edge cases. We know about these edge cases and that's just missing the point.
There are countless examples of emergent properties in these LLMs which by definition are solving tasks outside of its training set.
Its reasoning has been demonstrated on examples outside of its training set.
So I tried a word puzzle that would also require simple multiplication:
------------------------------
PROMPT: i am going to cycle 1600 miles, with 234 miles on gravel roads. on paved roads i will ride at 1929288282 millimeters per second but on gravel I will ride at 0.00000000202 parsecs per second. How long will the journey take?
-------------------------------
Now, I have to commend GPT on its ability to understand how you solve a problem like this, though that's not really very surprising given the huge numbers of such problems that exist in written materials. It precisely broke the problem down in a way that I suppose you could call "reasoning", but I would call "copying the formula for solving puzzles like this".
And how did it do with the actual math?
----------------
0.00000000202 parsecs per second is equivalent to 7499.6103827 miles per hour (mph), which we can calculate by converting parsecs to miles (1 parsec = 3.26 light-years = 19,173,511,840,000 miles) and dividing by the number of seconds in an hour:
0.00000000202 pc/s × 19,173,511,840,000 mi/pc ÷ 3600 s/hr = 7499.6103827 mph
----------
Utterly and completely wrong. Same issue with the millimeters per second computation.
It is completely obvious why LLMs cannot do this. They cannot perform even basic arithmetic reasoning, and even more fundamentally, the ONLY capability they have is to create likely responses to prompts. For some things, this is extraordinarily (and scarily) powerful. But it is not reasoning.
ChatGPT was trained specifically to solve "natural language understanding tasks".
You are missing the forest for the trees and ignoring everything else it's good at solving outside of its training set.
The emergent properties of neural networks should not be so casually dismissed.
You're arguing that since its not perfect at arithmetic reasoning then its not capable of any degree of reasoning in any domain. That is an oversimplification and just doesn't make logical sense.
I do not believe that any part of any current or future LLM (i.e. using the same fundamental architecture) is capable of reasoning, or in fact, capable of anything other than, essentially, doing a really, really, really good job of generating the next word in a response.
I am all about emergent properties of neural networks, but I absolutely do not believe that LLMs have them, specifically because of the way they are designed.
However, as to the specifics, people who seem to believe otherwise claim that they can reason, and so that's merely one specific angle of attack: to show that they cannot reason, and that in fact, everything they do is implicitly contained in their training set.
As I've already said, what they can do is enormously powerful and in many (most?) ways entirely unexpected, so I still regard the advent of LLMs as extremely significant, both from a practical but also a scientific point of view. I think it may force a revision in the most basic aspects of understanding human speech behavior, for example.
Nevertheless, I do not believe that anybody is served by believing that these systems can do things that they cannot. I do not understand why the seemingly magic results of these systems is leading so many into a denial of what they actually do.
But there is no world representation inside an LLM, only text (words, letters) representations, so nothing the LLM does can be based on reasoning in a traditional sense.
I would wager that if we build an LLM based on a training data set collection, and then we rebuild it with a heavily edited version of the data set that explicitly excludes certain significant areas of human discourse, the LLM will be severely impaired in its apparent ability to "reason" about anything connected with the excluded areas. That sounds as if it ought to surprise you, since you think they are capable of reasoning beyond the training set. It wouldn't surprise me at all, since I do not believe that it what they are doing.
LLMs contain a model of human speech (really text) behavior that is almost unimaginably more complex than anything we've built before. But by itself that doesn't mean very much with respect to general reasoning ability. The fact that LLMs can convince you otherwise points, to me, to the richness of the training data in suitable responses to almost any prompt,suitable, that is, for the purpose of persuading you that there is some kind of reasoning occuring. But there is not. The fact that neither you nor I can really build a model (hah!) of what the LLM is actually doing doesn't change that.
So what I'm saying is, GPT "knows" what a cat is. It "knows" what an orange is. It has inferred these concepts from the data set.
Imagine approaching someone who is tripping on LSD and demanding they immediately solve a 10 digit multiplication problem, then saying "AHA! You cannot solve it, therefore you are incapable of any reasoning whatsoever!"
Are you saying that NLP as a field of research did not exist before LLMs? This is a continuation of research that has been in progress for decades.
> But there is no world representation inside an LLM, only text (words, letters) representations, so nothing the LLM does can be based on reasoning in a traditional sense.
Not true. The model has learned a representation of semantic relationships between words and concepts at multiple levels of abstraction. That is the entire point. That's what is was trained to do.
It's a vast and deep neural network with a very high dimensional representation of the data. Those semantic/meaning relations are automatically learned and encoded in the model.
> It's a vast and deep neural network with a very high dimensional representation of the data.
the data is text, so ...
It's a vast and deep neural network with a very high dimensional representation of *text*
And yes, to some extent, text represents the world in interesting ways. But not adequately, IMO.
If you were an alien seeking to understand the earth, starting with humans' textual encoding thereof might be a palce to start. But its inadequacies would rapidly become evident, I claim, and you would realize that you need a "vast and deep representation" of the actual planet.
> Are you saying that NLP as a field of research did not exist before LLMs? This is a continuation of research that has been in progress for decades.
Of course I'm not saying that (the first sentence). Part of my whole point is that LLMs are to NLPs as rockets are to airplanes. They're fundamentally a "rip it up and start again" approach, that discards almost everything everyone knew about NLP. The results are astounding, but the connection with, yes, "traditional" NLP is tenuous.
Yes it is deep learning applied to NLP. Makes the old designs obsolete
> the data is text
It is not randomly generated text. There are patterns in that text. It was trained to model the semantics or "meaning" in the text. There is a structure in the text which the machine has recognized.
It automatically learned a model of many concepts without any of those concepts being explicitly programmed into it. That's the entire point of machine learning.
> But not adequately, IMO.
It is adequate for some things and not adequate for other things.
It seems that all you are saying is that GPT is not AGI and doesn't have human level of understanding and reasoning. No one disagrees with that.
No, I'm saying that LLMs perform what is essentially an amazing, really hard to understand party trick that happens to be so good it is actually useful. They may be indicative (surprisingly) of how a lot of actual human speech production happens (or may not), and if so, that's really good to discover. That this is surprising represents a bit of a failure of our imagination, but it also leaves untouched the parts of speech production that don't use this sort of process.
Like actual reasoning.
Answer:
"If the bib cannot see the bob due to being inside, it is likely that the bob, which is outside, also cannot see the bib. The reason for the bib not being able to see the bob is presumably due to some kind of barrier, such as a wall or a window, which would likely obstruct the view for both the bib and the bob."
Prompt:
"What type of reasoning did you just apply to solve that task?"
Answer:
"In solving the task, I applied abductive reasoning. Abductive reasoning involves inferring the most likely explanation for a set of observations or statements. In this case, I inferred that a barrier (e.g., a wall or window) is the most likely explanation for why the bib cannot see the bob, and thus, the bob likely cannot see the bib either."
ChatGPT solves the problem by a mechanical reasoning process, not a "party trick". It's a machine, not a person at a party. Surely you don't believe that it has seen that exact sequence of questions and replies in its training set and is simply regurgitating. You can argue about whether its actually using "abductive reasoning" or not, but that is besides the point that it involves some mechanical reasoning process over an interpretation of the prompt. It's not a simple regurgitation.
AlphaZero learned to play Go starting with nothing but the rules of the game. What is it regurgitating there?
> Like actual reasoning.
You're relying on intuition and personal beliefs of what constitutes "true" reasoning instead of formal rigorous mathematical definitions of reasoning. The general concept of reasoning includes what the language models are doing when they solve natural language understanding tasks, by definition.
It just sounds like a No True Scotsman fallacy.
> reasoning in a traditional sense.
We are talking about reasoning in a general sense. There are many types of reasoning in AI which I'm sure you know how to look up and read about. "Traditional" is not one of the categories.
Even when faced with evidence that contradicts your beliefs and proves that you are wrong?
LLMs are a type of neural network. What fundamentally prevents LLMs from having emergent abilities while other neural networks do have them?
How do you explain the emergent abilities that we have actually observed in LLMs?
The level of naievete around this stuff is quite incredible.
BEFORE: "We have no idea really what a system like this will do"
... amazing results ...
AFTER: "Wow, look at those emergent properties"
> The level of naievete around this stuff is quite incredible.
"Emergent properties" has a formal definition in the literature.
In machine learning we don't explicitly program the machine to understand anything. It automatically learns patterns in the data. That's the entire point of machine learning. With such a large neural network and training set obviously it's hard to predict all of its capabilities due to the sheer scale of it all. Of course we cannot predict exactly how it will model things.
Take this for example. No one programmed it to understand Go. It learned by itself and became the world champion. That's what deep learning is capable of.
https://en.wikipedia.org/wiki/AlphaGo_Zero
"The neural network initially knew nothing about Go beyond the rules."
Give it this prompt: "Write a short play where "Karen" (who behaves as the Karen meme) is on a romantic date with Hunter S. Thompson. They are eating at a Chinese restaurant."
It is able to script their interaction and dialog in a way that makes sense in the context of the setting and their personalities, including an absurd meme character. That is an emergent ability that it was not trained to do and is certainly not in its training set.
Try it. Add different characters. Ask it to rewrite the play using pirate metaphors. You can go deep into its "mind" and see the emergent abilities at play. Just apply some creativity and skip the boring arithmetic problems, as that's a well accepted weakness of this type of model.
LLMs are not good at math. But this is a subset of reasoning.
Chain-of-thought on logical inference tasks (using fake labels so we are outside the training set) shows they can do reasonably well at these.
Nevertheless, it's likely that the best approach for pure reasoning tasks will be to connect a LLM to a real inference engine (datalog or something) and rely on the LLM to perform the mapping to the inference engine inputs and outputs. This is similar to the "System 1" and "System 2" models of human thought.
Any human who has learned multiplication can do this. AFAIU, LLMs cannot unless the computation exists within the training set. They have zero arithmetic reasoning capability.
I just asked ChatGPT: "19191920 multipled by 10292111772"
It said:
--------------
To multiply 19191920 by 10292111772, you can use the standard long
multiplication method as follows:
19191920
x 10292111772
-------------
19191920000 (the product of 19191920 and 1)
153535360000 (the product of 19191920 and 8)
1535353600000 (the product of 19191920 and 2)
-------------
196837644266310240 (the final product)
Therefore, the result of multiplying 19191920 by 10292111772 is 196837644266310240.
----------------------------------This is completely wrong. It is not doing arithmetic, and it is not capable doing arithmetic.
"Reasoning" is a much broader term than you think it is.
Furthermore GPT-4 is able to multiple slightly shorter pairs of numbers, and experiments on smaller neural nets, such as one using a dataset consisting of 20% of all combinations of two four digit numbers, quickly learn to generalize to successfully multiply any four digit pair even though they aren't in the training set.
Try again, give an answer that actually works this time.
----------------------------------------------------------
PROMPT: I live in a location with heavy clay soil, and plan to build an adobe home. Will it be well-insulated?
RESPONSE (edited and truncated): Building an adobe home in an area with heavy clay soil can be a good option for creating a well-insulated home [... ] The insulation properties of an adobe home will depend on several factors, including the thickness of the walls, the quality of the materials used, and the construction techniques used. Adobe walls tend to be thick, which can help to provide good insulation properties. The high thermal mass of adobe can also help to regulate temperature and humidity inside the house, keeping it cool in the summer and warm in the winter.
-------------------------------------------------------
So, here it is clear that GPT is just regurgitating information contained in its training set. Why am I so confident? It is an extremely widespread misconception, repeated almost everywhere, that adobe homes are "well insulated", and almost everywhere this is written about, you will the linkage between high thermal mass and insulation.
But this is actually completely incorrect. Adobe is a very poor insulator - a typical wall has an R factor of around 2. Ask someone who knows and you will find that "adobe is not an insulator at all" (e.g. https://www.greenhomebuilding.com/QandA/adobe/mass.htm)
So I went back to GPT to see if it "knew" about the poor R value:
--------------------
PROMPT: what is the R value of an adobe wall
RESPONSE: The R-value of an adobe wall depends on several factors, including the thickness of the wall, the density of the material, and the specific mix of materials used in the construction. Generally speaking, adobe walls have an R-value of around 0.2 to 0.5 per inch of thickness, which is relatively low compared to some other building materials.
For example, a 12-inch thick adobe wall might have an R-value of around 2.4 to 6.0, depending on the specific mix of materials used. This means that an adobe home will typically require additional insulation, such as fiberglass batts or blown-in cellulose, to achieve a higher level of thermal performance.
-------------------------------------------------
OK, so it does "know" this, and what is has generated here is factually correct, but at odds with its initial response. If it "reasoned" it would know that, given this R value, an adobe wall is pretty terrible for insulation, and would have focused on the details in this response in answering my first prompt.
Then it continues:
-----------------------------------
However, it's important to note that the insulation properties of adobe walls are not solely determined by the R-value [...]
-----------------------------------
and this final claim is completely incorrect. Insulation properties are 100% represented by R values, and anyone who actually knows this stuff would know this. It then goes on to repeat the stuff about thermal mass, which is important for how a house feels, but unrelated to its level of insulation and thus its heating requirements etc.
Now, I imagine that given all this, one could do some prompt "engineering" to get GPT to spit out something that reflects the answer that a human who actually knew and could reason about this stuff might. But I have zero doubt that what you'd actually be doing is adjusting the vocabulary to make it more likely it would base its response on e.g. the Green Building Advisor article above. I do not believe there are any prompts, or anything else, in GPT or any other LLM, that will cause it to "reason" ... hmm, let's check the R value for adobe, nope that's pretty horrible, the house will not be well insulated unless you ....
Try this prompt: "Taking into account the r-value of adobe, I live in a location with heavy clay soil, and plan to build an adobe home. Will it be well-insulated?"
These edge case "gotchas" are missing the point.
The sort of logic systems that were the focus of a lot of AI work before "deep learning" came along would certainly have "taken the r-value of adobe" into account (had they been exposed to such knowledge). That's because they explicitly reason about things in the world that they are trained to reason about.
Gary Marcus has been quite usefully vocal about this. We used to try to build AI systems (some still are) based on the idea that you need a world model, and you need logic and inference and relationships.
LLMs have convinced, it seems, rather a lot of people that we can just discard all that - "the system will learn the patterns all by itself".
Marcus doesn't agree, and neither do I (not that my opinion is worth much).
Now, for all the disagreements in this thread I have not seen anyone claim it knows and reasons why things are the way they are, I don't disagree that it knows what to do for different inputs but it is not processing the input and making a reasonable decision more like mapping input with most approximate output. Is thay incorrect of me to say?
Yes it is actually modelling the meaning and context around it. ChatGPT is a ~100 layer deep neural network that was trained specifically to solve "natural language understanding tasks". The term "understanding" is an important term in NLP research along with the concept of "semantic similarity".
I believe it is more powerful and ultimately (eventually) more dangerous than you fear it is.
My point about mistakes is not that they were made but the way they were made indicates it was attempting to approximate. Someone mentioned a famous mathematician who got basic multiplication wrong, now if you are wrong because you missed some steps in the process that can be proven. But if your mistake is because you guessed the that can also proven by showing defects in your answer were arbitrary.
Pretend you're grading a student, you can tell when they guessed and got it wrong as opposed to tried to follow the process bur misunderstood something or make a critical error.
No, ChatGPT is not simply regurgitating. It actually understands the data. It is one of the largest deep neural networks ever constructed.
Consider AlphaZero, the neural net that learned how to play Go and became the world champion. It started with nothing but the rules of the game. Deep Learning systems build a generalized model and reason about it. They are not simply regurgitating what they have seen in their training set.
> These emergent abilities are not actually that, but a result of humans' poor understanding of cognition and communication.
My counterargument is that humans hallucinate too, and often. As just one small example, eyewitness testimony is stupifyingily unreliable. Neurological research and even basic behavioral research shows our brains act as bullshit machines fabricating satisfying narratives constantly. Not to even get into the fact that the word hallucination still has a non-AI meaning, and that dreams exist. As I put see it GPT models simply hallucinate more often and, more noticably, in a different manner than humans. The hallucination frequency need not reach zero, only human equivalent or better , and GPT-4 is already much better than GPT-3.
I agree with everything else you said fully. "True" reasoning machines or not, society will be catastrophically destabilized. Amongst the chaos I expect plentiful of "normal" conventional and nuclear war to go on.
Hallucinations are where you hear, see, smell, taste or feel things that appear to be real but only exist in your mind. Get medical help if you or someone else have hallucinations.
Why do we say LLMs hallucinate and why do people keep parroting the same thing "oh humans hallucinate", do we really hallucinate all the time? I can think of only one time a healthy adult hallucinates and it's not sitting around drinking a cup of tea.
It’s not about humans intentionally lying or truly hallucinating like on drugs. It’s about confidently thinking they are right about something which turns out not to be true.
I mean, if I say it like this is like humanity’s core business.
I think this: "It’s about confidently thinking they are right about something which turns out not to be true."
Unpopular opinion, most people who say this are actually not great listeners and don't take the time to understand peoples messages, so it sounds like "everyone is wrong or dumb", I find, nearly always, if I inquire more deeply, most peoples views or opinions are perfectly valid or they at least have good reasons for believing in something I would otherwise dismiss as false. So it's not that they're necessarily hallucinating, they're just got good reasons for having an alternate take on reality.
For an LLM, the truth is just an approximation of what it's been fed. I think for living creatures with past experience of their own, the ability to understand reality is more complex and nuanced and of course, includes their unique experiences.
My observation has been that nearly all intellectual views, opinions and "facts" are in some ways approximations. Wrong and and waiting to be revised at some stage.
Alexander Grothendieck, one of the greatest mathematicians of the 20th century, answered "57" when asked in a public lecture to provide an example of a prime number. 57 = 3 * 19 is not a prime number.
According to your argument, this would imply Grothendieck did not understand what a prime number is. Which is laughable.
human wrong => no problem, mistakes happen
Practically for outcomes it doesn’t matter if that learning is happening in the LLM or on a layer above it.
But 17, 37, 47, 67 and 97 are primes, so he probably applied a pattern subconsciously because you would never say 27, a drill cube, nor 77, which is divisible by 11 on sight.
I've taught mathematics at assorted levels, I think I can tell the reasoning behind quite a few human mistakes there. I analyse them, they're not usually just whatever, there's a wrong chain of thought behind the interesting ones. That's how I design exercises, tests and generally teach stuff to humans.
So I've been doing this mostly within Skype and the thing doesn't err because it's reasoning wrongly, it errs because it just comes up with stuff, there's nobody home. And that would be my professional assessment of it as a student. And yes, quite often it gives you good answers for the initial base line questions, but if you refine the problem to probe its level of "understanding" it's not difficult to derail it, it elaborates on its previous procedures while it's being steered towards a wall of nonsense.
https://en.wikipedia.org/wiki/Braitenberg_vehicle
Samuel Johnson: “Depend upon it, sir, when a man knows he is to be hanged in a fortnight, it concentrates his mind wonderfully.”
Also mathematics is not the right field to compare this to because there are rules. In languages and object recognition/synthesis it is all subjective. Understanding here means understanding context and human-subjective interpretation.
When I say "that sunset is beautiful" you understand what I mean, ML models simply approximate based on what they see other humans do or say. I am not calling the sunset beautiful because other people are, it is my own subjective interpretation.
This doesn't seem to make sense. If anything the opposite is true - if the things that are hallucinated make sense (even if not true) it means there is some "understanding" or a world model.
For example, you tell Dall-E to draw a picutre of a man smoking a pipe but then it draws the pipe coming out of the mans butt and instead of a head the man has a leg on his neck. This is approximation. Now if the pipe looked wrong or it was a trumpet or the guy's head looked weird but still a head maybe it knows what a head is and it just made a mistake and it knows that the pipe goes into the mouth so it will be somewhere near the face.
Undesrstanding is a 3yr old child drawing terribly, approximarion is drawing really well but all wrong.
All this applies to LLMs I used Dall-E because pictures are easier to talk about.
You can see this because of the big deal people made about image gen tools getting hands wrong: it was the most significant error that was systematically occuring.
An artist who knows how to draw hands and faces will never make that mistake, especially when self-correcting is so easy.
The only explanation is that it is approximating based on what it learned from the large swath of training data.
Kind of like if a human remembered answers to a multiplication table by memorizing every input,output and trend as opposed to knowing how to process the data within the rules of math and generate output. LLMs imo don't understand the rules of language and context they only approximate what a rule compliant system should output.
I've never seen significant or systematic errors with faces.
It sounds a little bit like you haven't actually tried these tools. The kinds of errors you seem to think they make just aren't there in practice. I'd encourage you to try them out!
> LLMs imo don't understand the rules of language and context they only approximate what a rule compliant system should output.
This is an area I know a lot about.
There are no real universal rules for English grammar. If you look at something like the Penn treebank you can see that English - as used by humans - is more exceptions than rules. The fact that LLMs outscore any rule based system merely means that our grammar rules are mostly things derived from how English is used in practice, not vice-versa.
These emergent abilities are not actually that, but a result of humans' poor understanding of cognition and communication.
What concerns me very much is how the harms that can be caused by LLMs has been so greatly under reported.
I imagine being someone with the right power and access telling ChatGPT "find all people that would vote against this candidate in real time and devise ad content and social media messaging and bit interaction to change their minds or discourage them from voting" heck, any intel org of a major country is probably already working on this. No more whistleblowing or posting anonymously on social media, companies would even share models based on private email and conversations you had so other companies could use LLMs to identify everything you posted elsewhere and to have LLMs designate a score for hoe hireable you are. Police can crack down on crime better but also crack down on dissent or any police reforms.
And we aren't even talking about war time use of LLMs or what happens when you marry something like ChatGPT with Dall-E and make it all real-time.
I am warning anyone who will listen. Smartphones are the most dangerous things out there. Any service or interaction that depends on them is deteimental to peace and liberty of the masses long term. People have not learned a thing from Snowden or 2016 elections.
And why are all the smart journalists asleep on the job on this topic. Where are the unreasonable scaremongerers when you need them!