Additionally, while we don't know how us humans functionally reason, it's believed that the predictive nature of our brain is central to our reasoning abilities. In some way, it's similar to the autoregressive of LLMs.
For a collection of emergent properties in LLMs, I recommend this paper [2]
[1]https://arxiv.org/abs/2201.11903.pdf [2]https://arxiv.org/pdf/2206.07682.pdf
A solid example is the tool use. Package up a brand new, never existed before or seen in the training data plug-in, give a few paragraphs of human readable documentation, and suddenly GPT-4 can interact with other systems, databases, and models. It doesn’t have 10,000 examples of interacting with this novel plugin to predict next word off of, it just has a short English description, enough for a human to be able to use and understand the same plugin effectively.
There’s a reason this is knocking the socks off many smart folks. It’s upending a lot of the sacred cows of human intelligence.
Most people are still talking about an article they read in the New York Times about GPT-3.5 Turbo. Not that 3.5 Turbo didn't exhibit some reasoning abilities, but they were more difficult to suss out. GPT-4 can still seem pretty vacant sometimes and finding "magic" in it has a lot to do with building up a "state" where the model is generating reason. It does not happen by default.
This will all be different in another month or two, as more people have access to the latest models and new models get released (assuming there is no agreement on a pause). Then there will be another lag as laypeople catch up.
There are scenarios that may play out where this speeds up even further and whole generations of AI completely fly under the radar of public acknowledgement.
Or this all collapses in a debris pile of hype. I feel like I am just along for the ride at this point, trying to keep my head above water. I reckon this is going to cause a tremendous amount of stress for a tremendous number of people.
So? What makes an LLM be incapable of reasoning in your view? It's like saying a boat is massive metal object so it cannot float!
You could easily argue any good language model would need to learn how to reason to be good at next token prediction. Reasoning therefore should, if anything, be an expected emergent property of a sufficiently large LLM.
The fact you can ask GPT problems that require reasoning and it gets them right frequently suggests (although admittedly doesn't prove) some degree of reasoning capability.
I think a lot of the annoyance felt around here is the tendency to apply human attributes to LLMs. It’s not to say that Chat GPT isn’t fulfilling the definition of “reasoning” on some level, even if it was found to fully meeting definition. I think it’s more the leap to the conclusion and that a lot of Hacker News readers are rankled by the leap lacking research and facts to back it up.
Anyway, it’s all really a philosophical discussion anyway since the definition of “intelligence” and “reason” are soft terms even when applied to human beings. We can usually hand wave it away as “intelligence is when humans do it the way humans do it”. Now that we’re on the cusp of creating actual artificial intelligence, I think we’re all finding those soft definitions cumbersome and are struggling to refine them as a society, which I think is a good thing. I also sometimes get rankled at someone asserting that some AI example has some human attribute, but mostly I think it’s an important part of that larger discourse about how we define these things, and that is a good thing in my mind.
To predict the next word very accurately for a huge variety of texts, a model must have some amount of real understanding. Sure, it may look different than human understanding.
But there’s no way around this. GPT could not predict so accurately if it didn’t have any understanding the world.
To at least some degree, maybe language is enough for both. Language doesn't just express concepts, but it contains them.
Let A_i be a set of functions that transform sequents in a truth-preserving manner (https://en.wikipedia.org/wiki/Sequent).
Do you have a reason for believing that a transformer couldn't out-perform existing techniques for choosing, given a sequence A_n(...A_1(Prop)...), what token comes next?
There are certain types of arguments that you could make, eg complexity theory does give us some bounds, but these models are large enough that (a) you can trade off enormous amounts of time for space, and (b) even time is questionable since forward passes result in truly enormous amounts of (CPU-indexed) computation.
And of course you can say hand-wavy things like "these models can only at best interpolate", but how many mathematics papers get rejected every year due to lack of novelty? And how much actually useful reasoning is novel, either in its particulars or especially in its generalized schematic form?
The true answer to the question of whether GPT-4 is a decent reasoning engine is, I think, close to your intuition: probably not. But in general, I don't see any good reason to believe that transformers can't do reasoning. In fact, quite the opposite.
If theory of computation has taught us anything, is that complexity very quickly outgrows everything that is even in the grasp of mathematics (e.g. busy beaver).
I just don’t think that a model that fundamentally can only take a fixed number of steps will ever make the jump. A human in theory can “go into a loop” and let’s say apply the same rules ad infinity, which is simply a must to most interesting problems. So - I guess GPT-4 can reason on the “trivial” degree simply by “caching” the output of the real algorithm, but that doesn’t scale at all.
True. But if the science/practice of computation has taught us anything, it is that many problems can be solved for all practical purposes even if they are undecidable or intractable for all theoretical purposes. E.g., the halting problem is undecidable but software verification/static analysis tools that check for termination exist and are even widely used in some industries.
> A human in theory can “go into a loop”
A computer can as well, by identifying loop invariants. And even the discovery of loop invariants can be thought of as a next-token-prediction problem, if characterized in as a sequence of states followed by the question "if x_1=t_1, ..., x_n=t_n, then what are these variables equal to at state N+1?"
> So - I guess GPT-4 can reason on the “trivial” degree simply by “caching” the output of the real algorithm, but that doesn’t scale at all.
I am more-or-less in agreement about GPT-4, but I think claims about the limitations of the architecture/problem statement are over-stated.
Why is treating GPT-4 as a reasoning machine so controversial? And by that, I don’t mean why it is wrong. I am asking why it upsets so many people.
We ignore a whole bunch of lies and half truths every day. What is it about GPT-4 that doesn’t let it get a free pass like UFOs or EmDrives?
What I am really interested in understanding is the possibility that at least some of these strongly dismissive responses are a bit of a threat response.
The other interesting possibility is that for over a decade, we got used to dismissing AI startups - and now, that habit could persist despite a real breakthrough.
Another possibility is that we were all given an overly simplified model/explanation of machine learning and now, those simplifications are being a real hurdle to comprehension.
It’s extremely fascinating to watch this play out. For my generation, this is the first time we’re seeing a brand new fundamental technology get deployed and alter society.
I think what annoys me about claims like this is two things, and it has to do with in-groups and out-groups.
1. I am concerned about how non-developers perceive us developers when they hear claims like this, since I expect that particular hype balloon to pop in the future.
2. I am distressed to see developers (who should know better) make these types of claims, especially without regard to 1.
If I were more into physics, I'd probably get more worked up about em-drives. As it is, I am enjoying LLMs and consider them an incredibly useful tool, possibly up there with things like GUIs and IDEs in terms of productivity enhancement. But it's still just extra-spicy autocomplete, and I think forgetting that could be dangerous.
> and now, that habit could persist despite a real breakthrough
Or the breakthrough has a local maximum, because if LLMs are just statistical token predictors without any real reasoning, it will never achieve AGI level of AI.
Unless, of course, we redefine what an AGI is.
Human brains are known to create "meaning" where there is none and with a perfectly created sentence they will assign meaning (reasoning in this case) where there is none.
One week ago there was a case of a man (in Belgium) who "sacrificed" himself because the AI chatBot convinced him that there was no solution to Enviornment problem. He assigned meaning ("Even the AI could not find a way out") where the chatbot was just parroting what it was reading in the media. Note that in this case both the general media and the AI give the same "result"=no hope, but as long as it was in the general media, he was still hopeful that humans just couldn't find it whereas the AI convinced him.
You may diregard this as an extreme example but we have the general tendency to attribute correctnes to everything "computable". If I can trust the calculator that was created to do math why should I not trust the AI that is created to do Reasoning? People need to be aware that there is no reasoning behind it.
At the risk of attracting down votes, you could ask the same question about the visceral responses to crypto currencies you see here.
It's certainly true they have their problems. Bitcoins 7 transactions per second makes it's chain a no-starter for most uses, the numerous multi million dollar losses due to bugs in Ethereum contracts should scare anybody. But the reaction you see here goes well beyond that. After all they continue exist, even performing the occasional useful task, and they are gradually (glacially?) overcoming their initial shortcomings.
The aversion to everything crypto seems to be more of a guttural reaction to the existence of something that uses algorithms to generate and distribute trust. It challenges the human social trust networks we've been using to do the job for centuries. This aversion to calling the obvious deductive capabilities AI's "thinking" or "intelligence" looks to be similar. I guess it challenges what has been a uniquely human ability.
"Next token prediction" taken to a certain mathematical limit implies reasoning.
> Suppose I have a cabbage, a goat and a lion, and I need to get them across a river. I have a boat that can only carry myself and a single other item. I am not allowed to leave the cabbage and lion alone together, and I am not allowed to leave the lion and goat alone together. How can I safely get all three across?
I just asked ChatGPT v4 this question, assuming it would have been fixed by RHLF training but it spat out step 1 as “First, take the goat across the river and leave it on the other side,” leaving the lion and cabbage alone against explicit directions in the riddle.
It clearly can’t reason, just produce a convincing imitation most of the time
No, the reasoning is the prior data. Thought experiment: GPT-4, but the training data contains no puzzles of this kind. How well would it do?
> Try it again, but this time make up nonsense words for the three items.
Seconded. I also would like to see the results of this experiment.
>> Suppose I have a blurple, a mantiple and a hoje, and I need to get them across a river. I have a boat that can only carry myself and a single other item. I am not allowed to leave the blurple and hoje alone together, and I am not allowed to leave the hoje and mantiple alone together. How can I safely get all three across?
> This is a classic river crossing puzzle. Here's one possible solution:
> 1. Take the blurple across the river, leaving the mantiple and hoje behind.
> 2. Leave the blurple on the other side of the river and return to the original side.
> 3. Take the hoje across the river and leave it there with the blurple.
> 4. Take the mantiple across the river and leave it with the hoje.
> 5. Return to the original side of the river and retrieve the blurple.
> 6. Finally, take the blurple across the river to join the mantiple and hoje on the other side.
> This solution ensures that no two items that cannot be left alone together are ever left alone together.
Of course, my second stipulation was "I am not allowed to leave the hoje and mantiple alone together" and yet step 1 says "leaving the mantiple and hoje behind"
So no, despite using the words "classic river crossing puzzle," it clearly expressed no understanding of the puzzle.
That really, really, really demonstrates something. I'm not sure what.
ChatGPT is an implementation of the Chinese Room Thought Experiment.[0] Our pareidolia allows us to ascribe reasoning faculty.[1] But it is far from proven.
For this whole discussion, it actually really does matter if GPT-4 is able to reason or not. That is the essence of the discussion. It's not whether or not it looks like it can reason.
My own car includes cameras and can keep me from disaster by applying brakes and correcting steering, but it makes no claims whatsoever to be self-driving.
A Tesla car unfortunately looks like self-driving, walks like self-driving, and quacks like self-driving, but it turns it does matter how much the nitty-gritty details prove the thing is absolutely not actually self-driving, in that encounters with things outside of its training model seem to result in collision or decapitations.
It is the trust people place in their cars that results in disaster when they reach the edge of what the cars can handle. They aren't ready to take over the controls and understand their current situation in a fraction of a second.
Placing too much trust in LLMs could have the same effect. I've run into several situations where I asked Chat-GPT technical questions and it confidently gave me detailed answers that clearly represented absolute failure. Fortunately, I don't trust it blindly, or at all, so I wasn't fooled.
I'd prefer an LLM more like my current car, that didn't try to do too much, but that's just not in the nature of how they operate, so instead we're all one emergency vehicle parked across the street away from a terrible accident.
The point of my comment is that depending on how you interpret the description "predicts the next token", something that can do this well enough should be superhumanly clever. The task is "AI Complete".
Don't misinterpret me as saying that GPT is superhumanly clever. I'm just saying that something that's sufficiently good at what people think GPT is doing, should be superhuman.
I gave ChatGPT 5 questions from a verbal reasoning test on sequence of words and it got 2/5 right.
Here is sample question:
Arrange the words given below in a meaningful sequence.
1. Poverty 2. Population 3. Death 4. Unemployment 5. Disease A) 2, 3, 4, 5, 1 B) 3, 4, 2, 5, 1 C) 2, 4, 1, 5, 3 D) 1, 2, 3, 4, 5
ChatGPT>
B) 3, 4, 2, 5, 1
Unemployment can lead to poverty, which can lead to disease. Disease can lead to death. The population of an area is affected by all of these factors. Therefore, the correct order is 3, 4, 2, 5, 1.
Correct answer was C
I would expect if it was effective at reasoning it should be strongest on verbal associations. And while it is possible that enhancements to future language models might provided them with better reasoning ability I don't believe you can simply push the existing design to that limit.
The poor results from the verbal reasoning tests demonstrate that reasoning is more complicated than what a transformer based language model is capable of.
Overall, the argument is moot because the statement made in the article was it was a reasoning engine which is not equivalent to being capable of reasoning.
"Machined" means a precise implementation. A machined reasoning engine would be precise, accurate, and consistent. GPT-4 is impressive autocomplete, but not what we would expect for machined reasoning.
On one hand, it makes this topic difficult to follow since stuff aren't clearly defined anymore and to software folks, I assume this is a major problem. On the other hand, it is fascinating how this has actually resembled biology where literally everything is poorly documented and for every rule, there is a glaring exception or flagrant violation of it, existing simply just to spite that rule and our understanding of it.
I disagree with this conclusion, which some supporters of GPT-4 being a reasoning engine might find distressing.
The AI effect is a real thing. And even real humans had been considered unable to "reason" once upon a time. I will give everything the benefits of the doubt concerning intelligence and sentience. Because history has shown how terrible it would be to get a false negative.
The point I'm making is not that GPT is as intelligent as a brain, but rather that the ability to produce the next token in a way that forms arbitrarily intelligent responses to arbitrary questions, may well require something resembling real, actual intelligence.
You decide which comments to do this for as well.
How does GPT decide such things?
All of them? If not, how to determine which ones?
There is a difference here between you and the machine
Looking at the LLM algorithms, they do appear to be next token predictors. Why do you ascribe something more to them when the algorithm is known?
Given we know that causal patterns are in the training data, and that LLMs are capable of embedding algebra, it is understandable that they appear to "reason", while we as humans can also describe the algorithm and mechanics actually at work here.
How does it determine how to respond to any prompt? Exactly. It might reasonably predict responses to those comments that, according to its black box, are somehow most likely to receive a reply. Different GPT instances prompted or trained to have different personalities would choose differently based on their priors and biases.
I'm pretty sure the vast majority of human beings is able to reason without really having much idea at all as to how exactly they do it. Educated humans can use language to talk about reasoning in terms of logic, and even make logic engines out of doped silicon and whatnot, but as of now, none of us has any idea of how reasoning is implemented in the mass of fat and protein that's inside our heads.
Yet another case, many times we will respond (or not) to a questions with another question, to elicit deeper considerations. In my experience, GPTs will answer any question without thought, pun intended :]
The original question was, given that there were X comments which could be replied to on this page, how and why did the poster decide to respond to the one to which they responded, rather than either "all" (like ChatGPT would) or "none?"
If you mean feeding it the messages one at a time, then, sure, it will respond to each of them because that's what its default "goal function" is. (I assume that we're talking about a single "conversation" here so the context is retained – feeding it each comment out-of-context would be very unfair.)
If you prompt it to only reply to messages it considers "interesting" in some way, and return "I'll skip this" for the others, it would certainly do that.
Yes, GPT is code that only runs when you, erm… run it. It’s not an autonomous being that’s cut loose on the internet and given free reign to use as much compute resources as it wants, a la skynet, replying to random forum posts. But nobody’s really saying that it is, so it seems a strange argument to say “it’s just a next token predictor”, if by that you really mean “it only runs when you give it a prompt.”
The modality of GPT is that it responds to prompts, but the interesting question is “are the responses given a sign of intelligence?”. Saying “no, it’s just a next token predictor” is IMO a sign of a motte-and-bailey fallacy. Your motte is that it only works in response to prompts (trivially true and uninteresting), and your bailey is that it has no intelligence (very debatable and IMO false.)
GPT has no "reason" behind its words. Its actions are not tied to the general betterment of humankind. When we all sit here with our coffees and donuts (and farting) we're generally trying to push others to see our worldview so that maybe, just maybe, the world can change a little bit.
If we eventually have a way to tie an AI to all these moods (hunger, pain, tiredness) and all the other things of the human experience, it would ground the thoughts and text it generates at some point such that it would want to DO things it wasn't doing before. It also can't right now anyway, unless we give it that ability (ReAct pattern could).
We can, of course, simulate all these human experiences, with "best practices" but then the reason won't be really tied to human reasoning. It will be just more text autocomplete with additional instructions.
Sure, LLMs are not inherently goal-oriented. But that doesn't mean they fundamentally cannot exhibit goal-oriented behavior if either prompted or trained to do so. And if we remove the "one-off" restriction and allow the retention of some sort of an efficiently encoded state between invocations, who knows what might happen?
So you saw the parent comment about hype and clicked into this comment box without a thought of what you might say other than "When?" Really?
Did you know whether you agreed or disagreed with the parent comment, or did it take until the word "reductive" for you to firm that up?
If you're saying that the part of your brain that controls your fingers is doing so one word at a time, then sure, obviously that's true. But if you're making the opposite claim, that LLMs like Chat-GPT are "deciding" what to say and only then working out the finer details of how to express that decision, that's a claim I'd like to see explicitly.
If you are suggesting that your approach to intelligence is parallel to that of an LLM, then either you're arguing for something far more radical than most as an emergent property of LLMs, or you're suggesting that you only form opinions one word at a time as you type, which I think would make you, well, I won't insult you, but I think you're probably doing a little more than that.
I just don't think this is it, although I freely admit that I'm amazing at the technology.
As it stands, Chat-GPT is probably better at giving the appearance of reasoning that some percentage of humans. That reflects poorly on those humans, but I've certainly known people in my life who seemed like they rarely knew the end of a sentence when they began speaking.
However, the focus of LLMs on text-prediction is not a path that leads toward human-like reasoning, just the appearance thereof. If and when we have AIs that actually do reasoning, it will be some other branch of AI, possibly also incorporating an LLM for communication, but not based solely on LLMs.
Imagine I lock you in a room filled with french-language magazines and movies and books. Having nothing better to do, you watch and read them (as well as you can). Periodically, I have my confederate, a french man, enter the room and speak a french sentence with one word missing. He asks you to fill in the missing word.
We keep you locked in this room until you've achieved some target benchmark of correctly saying the missing word, say 20% of the time.
Are you then, just a "next-french-word-predicting" robot?
No, obviously not. To have achieved the task and escaped the room, you will have had to have gained a significant understanding of French.
It is the same with ChatGPT.
No, in this scenario the posited person has only gained a significant understanding of patterns of French language, not of the language itself. You may believe that difference to be meaningless but tangibly it is the difference between identifying the next shape in a sequence and understanding that there is some larger purpose at play (context, if nothing else).
What actual understanding of the language they may have gained comes from second order effects -- seeing the language used in a movie for example is going to drastically influence and impart meaning that isn't present in the words alone.
This is somewhat born out in how ChatGPT and other LLMs being hyped are often really aggressively wrong about things. They don't get the correct answer in many circumstances, even when it's obvious to a human. If it truly understood everything it could from what it has been trained on, those gaps would not occur.
I don't think that's true at all. To explain my thinking, let's assume that I am an American, and therefore according to popular understanding, completely incapable of learning any language other than English. I despair! But then I think no, wait, I happen to have a gift for numbers. Like, my mental arithmetic stunned teachers from a young age, and I currently hold the world record for reciting digits of pi from memory. So what I will do, I think, is assign a number to every word I read or hear and where possible, visualize where that chain of numbers exists among the decimal digits of pi.
At some point the person would come in and utter some words I couldn't comprehend, but I would think, aha! That was almost 498153217845632754, except it was missing the second seven, which mapped to "contraire." And I'd do that at least 20% of the time, escaping the room with no understanding of French at all.
If I could map between memorized numbers and words quickly enough, I'd even be able to converse in French given suitable trigger words with a shred of understanding.
Because that's easy for a human, and the mathematical approach you describe is something like impossible.
Even for a machine, the mathematical approach is going to be impossible in practical terms as soon as input gets longer than a few sentences. To build any sort of probability table conditional on more than just a small number of tokens becomes astronomically big.
The intuition I'm trying (not very successfully) to get across is if you did this with a human, the human would end up learning French.
So Consider that as an existence proof that even if you "train to predict" you can still end up with actual understanding.
Beyond that, (and while this is less certain), probably to be able to predict well w/ any accuracy for texts of non-trivial length, you have to be able to understand; probably this is the only way to "compress" the insanely large conditional probabilities.
This is the mistake people keep making, literally anthropomorphizing LLMs! Because humans would find it easier to just learn the meanings of French words than regurgitate from a gigantic table of memoized tokens, these results must reflect understanding!
But of course, it's far, far easier for a computer to index a large number of tokens without any understanding at all than it would be for a person. Computers are not people. They have different strengths and weaknesses, and it is a mistake to forget that. (Ironically, one of our human weaknesses is also a strength: our pattern-matching skills are so strong we sometimes see Jesus in toast, or a face on Mars, when neither are actually there!)
What I described is literally how LLMs works. There is open source code to examine, and what it does is essentially what I said: tokenize words and store multi-word strings and identify their prevalence in training data, where the training data is an incredibly large corpus.
The experiment is about finding one missing word in a sentence, but the same logic is true even for long responses, or as I suggested, conversations.
To over-simplify: humans think in words, computers think in math. But it turns out that if you assigned a number to each word and parse enough of them, you can fake an understanding of words surprisingly well, even though it's still math.
I've given example elsewhere in this thread and on previous ones about the March 14 Chat-GPT giving incontrovertibly false information, even repeatedly in the face of correction, or seeming to correctly identify a popular logic puzzle while breaking the rules of the puzzle three times, then seeming to claim it had followed the rules of the puzzle.
> So Consider that as an existence proof that even if you "train to predict" you can still end up with actual understanding.
As a human, that is often true. Humans have trouble with arbitrary data, and it helps us to connect new data to existing data. In addition, we generally have a curiosity that prompts us to pursue new knowledge and meaning wherever possible, even when it doesn't actually exist. But if you're trying to suggest that because it is possible that humans can develop understanding while trying to only "train to predict," therefore it is probable that computers will develop understanding along the same lines, I suggest that's about three leaps too far.
ChatGPT is a neural network, a design inspired (loosely) by the human brain. How exactly it works is still not known. Like of course the raw details of how the computations work is known, but why the particular trained weights are what they are, and how the thinking actually happens is not known (so far as I know).
But I think it must be more than just something like looking up a conditional probability table -- calculate out the size of a conditional probabiltty of 8000 tokens or whatever and it'll be way way way more than the number of parameters of the model.
And no doubt as it stands ChatGPT is still far inferior to human intellect.
But I would note that making mistakes doesn't mean something like thinking isn't happening -- humans make mistakes all the time, often very dumb ones!
If you lock a reasoning being in a room, then regardless of your further actions with the French language, you'll be dealing with a reasoning being. Your sequence "you will have had to have gained a significant understanding of French" is deprecated on the fact that "understanding" is attributed to the reasoning being capable of it, not to the fact that they were locked in the room and forced to repeat the words.
And this is all that GPT is doing.
To do it it, you need a form of compression: reasoning, intelligence, knowledge.
We already know LLMs tokenize words, so they have definitely generated and stored such a table. The marvelous step forward from previous word-based solutions is that they store not just single words, not just pairs of words, but larger sequences of words, even up to short sentence length. It makes the table pretty large, as you suggest, but that's exactly we've been seeing, and each generation of GPT is that much larger than the one before.
In your analogy, the person never learned the meanings of the French words. Just how they usually go together.
So am I, maybe
One of the main lessons this whole GPT fiasco has taught us is that if you make a super-duper fancy autocomplete trained on terabytes of textual data, it develops some sort of internal logic/understanding. That is very exciting to me.
It is truly impressive.
My argument is nothing against it as a technical achievement, and everything against the sophistry that it is a reasoning engine.
This is very different from real reasoning that has unlimited (yet imperfect) memory.
Hey, that was 15 words, and I had them all in my head at once before I started typing! I was able to form a sentence of more than seven words all by myself!
You seem to be misapplying the concept of working memory, where seven (plus or minus two) is about arbitrary and unrelated words or numbers, not "items." In fact, it's generally seven for numbers, but lower for words. And again, that's unrelated words, like "person woman man camera tv," but probably even more unrelated.
GPT-4 is not a simple Markov chain.
We should be curious about how it does what it does when it seems to be reasoning.
More: https://skybrian.substack.com/p/dont-settle-for-a-superficia...
God of the gaps, reworked. Admittedly, a new claim to me. $10.05.
It’s not clear what the basis for your argument is.
It performs no reasoning, and instead predicts the next token
GPT-4 is a generative algorithm and large language model. It performs no reasoning, and instead predicts the next token in language. This works okay on average, because most data on the internet has some type of coherency to it, but there is a major gap between "reasoning" and "next token prediction."