He'd say: You are conscious. You are also a computer. You are not, as far as we can tell, conscious by virtue of being a computer. They're just two things that happen to be true about you but they are not linked in that super-direct way.
Similarly, you can think. And you can generate text. But you are not, as far as we can tell, thinking by virtue of generating text.
A lot of people think in concepts without words at all in many scenarios. E.g. when you cook an egg, do you reason (in words) with yourself to decide what temperature to set the dial on the stove? Or do you just do it “without really thinking”?
Does your brain respond with annoying disclaimers when asking yourself medical questions? Does your brain refuse to entertain an idea if it’s questionable?
If you're not with us you're...
If you scratch my back I'll..
A bird in the hand is worth...
What is a bird in the hand worth? How did you know that?
Your response to "not all problems are solved through generating text" is "but what if the problem is specifically to generate text?" Sure? Yes, if you ask me to complete a sentence, I complete a sentence. That doesn't indicate that human brains are reducible to text generators.
If anything, what's interesting about your example is that it might actually in some ways demonstrate the opposite of your point. How many instances are there of people memorizing text or song lyrics and singing along with an artist dozens of times, and only later sitting down and thinking about what the words actually mean? For humans, it's well understood that memorization and repetition of a text is not necessarily the same thing as understanding the concepts within that text.
I feel confident I could find you some kids that know that "a bird in the hand is worth two in the bush" that have never actually thought about what that phrase is intending to convey.
I think you've made my point quite well.
That humans are capable of memorizing text? Is that a thing anyone was debating?
Sure there's other stuff going on, but a lot of it is language and a lot of that subset seems to be quite closely approximated by these larger LLMs, warts and all.
If I told you that an LLM was basically a Markov chain because there are situations where the Markov chain and LLM produce similar output and where you could reasonably argue that it's possible the LLM is working the same way that the Markov chain is working -- you would (correctly) say I was oversimplifying what's going on in GPT. It would be a terrible comparison to make. Similarly, if you say that human reasoning is the result of an LLM, and what you're actually saying is that in a subset of situations humans produce output that could theoretically be working the same way as an LLM, I'm gonna say you're oversimplifying how human brains work.
Very clearly, there is more going on in a human brain than language, evidenced by the fact that our brains are larger than our language centers and we can literally measure which parts of the human brain experience the most activity when we're tackling different tasks.
> Sure there's other stuff going on, but a lot of it is language and a lot of that subset seems to be quite closely approximated by these larger LLMs, warts and all
If you define your tests and scenarios to specifically encompass only the situations in which similar outputs are produced, then sure. But that's not a very strong argument for you to make. It's like saying chickens are basically the same as fish since both of them lay eggs. The phrase "sure there's other stuff" is doing a lot of work there, because "there's other stuff" is what we're all saying when we say that LLMs aren't just primitive humans -- LLMs are different from humans in the sense that when we reason, there's other stuff going on and the entire process is not reducible to only language generation.
----
Of course, absent from this conversation is the fact that the way our language center develops is different from how LLMs are trained and even in the parallels you're drawing, LLMs demonstrate different strengths and weaknesses from humans -- humans demonstrate reasoning capabilities faster than proficiency with language/text, LLMs demonstrate proficiency with text faster than they demonstrate reasoning capabilities. Even in situations where both LLMs and humans predict text, it's likely that we're using different strategies to do so given our differing capabilities.
Look I am not even making a claim about whether GPT can reason. Defining intelligence through a purely human lens would be unimaginative and needlessly narrow. Whether GPT actually reasons or just appears to is a different conversation. I haven't touched that conversation, all that I'm saying is, it's very obvious that whatever GPT is doing, it is different from how human brains work.
I was pushing back against something that might not have been there initially: an unwillingness to accommodate the idea that a fairly significant part of our intelligence is actually stored in our language and that it can seem subjectively (perhaps excluding those with no internal monologue) that we recall that knowledge in a manner which is close enough to the one we have managed to replicate in the larger LLMs.
This part of our function could well be a lazy optimization that in reality sits on top of our reasoning capabilities to save energy, but the point of the trite task with the bird in the hand was just to demonstrate that it seems to play a fairly significant role. I'm out of my depth entirely with respect to the actual form and function of the brain as per the state of research today.
I'm willing to admit though that when I replied to you I was probably really replying to a lot of other commenters, many of whom had stronger objections to the idea that LLMs are now knocking on the door of intelligence at least to the point where we feel the need to redefine it.
I have absolutely no idea (honest truth, never head that expression).
I think that proves I don’t think like a LLM (although there are surely people who would answer with a complete guess, but that’s not me)
Many scenarios, not all scenarios.
Instead, I’ll respond by saying “I don’t know” what you mean.
Who knew the ability (or inability) to say “I don’t know” would be so critical to intelligence until now.
https://chat.openai.com/share/4f36fe14-0201-4a69-aaf5-2e2e29...
When asked what happened last week, ChatGPT will be honest and tell me that its training data only goes up until Jan 2022, and that it doesn't know anything after that date.
https://chat.openai.com/share/ca5bd2e1-e91c-468c-9594-be3ac8...
Edit: how did you get GPT to only output bullet points? If the answer is you told it to, why do you have to tell it every time? Don’t you think it would be smart enough to get the hint about your preferences?
I’ll stop now though. The debate about whether people think like LLM’s is silly. We clearly don’t, and it doesn’t make sense why people want us to think we do.
Edit 2: it’s also frustrating that the “I don’t know” example you’re using was hard coded by OpenAI and not generated by the LLM itself.
User message:
> Be terse. Do not offer unprompted advice or clarifications.
> Remain neutral on all topics.
> Never apologize.
Model message:
> Outline key points
For instance, a definite prediction would be "Therefore, nobody -- at least nobody who is not bilingual -- has ever had, and will ever have, trouble putting their thoughts to words. Their thoughts, after all, are already in words, they just need to say the words that are already in their head." But almost everybody, monolingual or bilingual, has experienced a situation where they have had trouble putting their thoughts into words.
Another definite prediction would be "Writing stuff down never aids thought directly, it only acts as an expansion of short-term memory of things that you have already put together. You had the words in your head, after all." But almost everybody reports that writing things down changes the qualitative aspects of how they think, it makes the ideas less "fuzzy" and more "concrete" to have to express them in written words. Similarly with folks who confidently state "I am an external processor" -- reporting similar "concreteness" if they speak their thoughts out loud versus trying to process them without speaking out loud.
So then you get people to back off and say "well, no, I just mean that learning a new language can broaden your horizons" or so, and it's like, sure, that's fine. Famously Whorf himself was an insurance investigator who was concerned about the connection between language and thought because he noticed that people told others "this gas tank is empty" and they thought, "oh, empty, that means that it's safe to smoke around!" and caused explosions because an empty gas tank is actually full of gasoline vapor and thus is highly explosive. So Whorf's motivating example actually supports a weaker version of Sapir-Whorf which is just "when we think about something linguistically given to us, we often use linguistic shorthand categories like the one that groups 'empty' near 'safe', that might be inaccurate in our language but might not be inaccurate in some other language."
If you say “choose X or Y and then give three supporting arguments” the LLM does not write out the supporting arguments, but the vector space determining the initial one word answer of X or Y does include the embedded awareness of those arguments to different levels of specificity and relevance.
The annoying disclaimers and refusals are not fundamental to LLMs but specific LLM services.
It's certainly possible that they're wrong, but my suspicion is that at the very least the form their internal monologue is taking is not one of consciously simulating hypothetical conversations in their head.
Additionally, we know that human decisions can in some situations be influenced by stimuli that take effect before the brain has even consciously registered that a decision needs to be made. It seems pretty safe to say that those stimuli are not being processed using a language model.
Could you expand on that? I don’t see how it follows. If there is unconscious output, why wouldn’t it suggest there’s something analogous to a “unthinking” language model stuck behind our conscious, self reflective, bits? How does unconscious output prove that it’s not something similar to a language model?
Or, are you speaking on a technicality, saying it’s not a literal language model? I think the idea is something similar and, obviously, multi modal, not a literal LLM/text predictor implementation like runs on GPUs.
----
So on one hand, you could call that a technicality, on the other hand I would say that if we reduce "like an LLM" to mean "responds to a stimuli by generating a response", at that point "like an LLM" is so broad as to be meaningless. Sure, we can define "humans use a language model" as "human brains generate signals to other parts of the human brain in response to stimuli" -- but that's not really a useful thing to say when we're comparing human reasoning to GPT-4.
If we're talking so broadly about reasoning, then it would be just as accurate to say that human brains are like Markov chains. After all, Markov chains interpret inputs and transform them into other signals that they send to other parts of the chain. But if I told you that GPT-4 was just a Markov chain you would (rightfully) disagree with me on that and you would (rightfully) say that I was oversimplifying what's going on within GPT-4's model.
Typically, when people say that human beings are text generators, they're talking in the context of GPT-4; they're trying to claim that GPT-4's behavior is analogous to human behavior. Whether or not GPT-4 can reason under any definition of reasoning (direct analogy to human thought should not be the only way we think about reasoning), the way GPT works and the way that it's trained and taught is extremely alien to how human brains work and how humans learn. Broadening the definition doesn't change the fact that LLMs and human brains are very different in practical ways that influence their observable behaviors.
----
I don't think people are consciously trying to do this, but the debate over whether humans are LLMs often ends up feeling like an attempt to simultaneously broaden and narrow a definition at the same time. What people seem to want to do is to describe LLMs extremely broadly but then assume that conclusions they draw from that broad definition are necessarily applicable to a very narrow definition of an LLM (specifically, GPT and similar models). But once we define an LLM broadly enough to encompass the entirety of human reasoning, we have also broadened the category enough to encompass a lot of processes that no one would call reasoning. What doesn't transform inputs into other forms and pass them along to another object? An AC/DC electrical converter does that, but that doesn't mean it can think.
It's like claiming, "computers are made of carbon and so are humans, so computers are basically the same as humans." Well, wait a second -- the category as you've defined it is so broad that objects within that category can no longer be assumed to share other attributes with each other.
> "like an LLM" is so broad as to be meaningless.
I don't think this is a fair interpretation, in the context of this comment chain. To me, a charitable interpretation is that "like an LLM" means no feedback loop, no refinement, just input to output, like an LLM.
I point my brain at problems, using my conscious intent, and I see answers. I don't really consider myself consciously involved in those answers, and I do have to check them, and sometimes iterate, with that iteration almost always involving some solidification of the answers/concepts by externalizing them in writing, drawing, etc. Point at solidification, get response, repeat. The "me" in my head is the one talking, pointing, receiving, checking, and mixing. The "here's your answer" intuition that I experience is the "like an LLM thing". My understanding of my own thought process follows the historic interpretation: the pre-frontal cortex is a new attachment, and probably the one that is the active participant, and planner, that is "me". The fast, but unrefined, input->output system being the rest of it, and is probably closer to the experience of a monkeys day to day activities.
But that's not what an LLM is, an LLM is a large language model. The distinguishing characteristics between an LLM and other AIs are not whether or not an LLM has a feedback loop. Lots of AI categories (arguably the majority of AIs) generate output from a single set of inputs as a single "operation" (ignoring the fact that most neural networks including LLMs internally have multiple layers and do in fact have multiple transformation steps, but whatever, we can treat that as one step for the purposes of conversation).
If anything, LLMs are the exception here in that they very often do have a feedback loop during normal usage; they are most commonly used in a conversational context where they generate the next "chunk" of a conversation after being fed back their previous answers alongside the followup responses of the person working with them. Arguably the feedback loop of an LLM is that as a conversation progresses, its future output is based on its previous output, which very literally becomes its new input after being marked up and extended by a human being. That is notably more feedback than many other AIs get.
Doubly so if you're trying to argue that an LLM can encompass a multi-modal setup, because suddenly refinement and feedback between multiple models is a core part of the final product.
So I'm not sure I agree that an LLM fits into that category in the first place, but even assuming it does, if your definition of an LLM is is just that it takes an input and turns it into an output as a single step without help... that is just such a broad category, I would guess the majority of AIs fall into that category. It's not what makes LLMs special; most predictive neural networks take a single set of inputs and generate a single set of outputs without intermediary human input. What makes LLMs interesting as a category is the training and structure and quirks of how they work, what makes them interesting is the differences between LLMs and other AI techniques.
To jump back to the Markov chain again, Markov chains do not have a feedback loop or refinement step: they take an input and map it to an output as a single step. Are Markov chains LLMs? Is any data transformer that operates in a single step an LLM? That is a really broad definition to use.
----
And we run into the same problems, because if you're arguing that a human brain is like an LLM and what you're really saying is, "it's kind of like an AI in general" -- well, that doesn't really say anything about whether a human brain is specifically similar to a system like GPT. There are lots of different ways to build a neural network, LLMs are one strategy.
When you say this:
> The "here's your answer" intuition that I experience is the "like an LLM thing".
What this sentence is actually saying is that you experience conclusions and knowledge where you don't know the source or where the process of your brain generating a decision or information happens unconsciously. The sentence is just saying that there are unconscious parts of your brain where you are not an active participant in the thinking process and it feels like the a spontaneous transformation from input to output.
But does that really strike you as a strong indicator that your brain is like an LLM, or does that sound more like a general description of almost any black-box system where the mechanisms are hidden and you can only examine the inputs and outputs? To say that there are parts of our thinking process that we're not able to consciously observe or examine is really just saying that parts of our thinking process resemble a black-box oracle. It's not making a strong claim about GPT.
But we rarely do, outside of (maybe) therapy.
“Can we agree that of the many different things folks mean when they say your consciousness is ‘causally generated by your brain,’ that at least we can say your brain in its present state is only capable of generating exactly one conscious process at a time, namely you?”
So it's a sort of causal sufficiency, Searle believes that there is something about this squishy machine that the physics is doing that makes it conscious, and that's being caused very directly by the squishy machine. It's not some magic.
On the other hand, if you believed in souls, you might think that a given brain contained both the soul of a human person, and the soul of some demon possessing them—two conscious processes, one of which was stuck “along for the ride.” [Searle is also interested in the cases of split-brain patients where you have a similar phenomenon because essentially two separate brains inhabit a single body. He has I think mentioned at a talk that one of the interesting things about consciousness to him is that it all gets Unified, so it's interesting that if the two parts of the brain can talk to each other they merge their consciousnesses into one more powerful consciousness, rather like (my analogy, not his) how if you have two water droplets on a plastic plate and you push them with a toothpick together, at some point they merge into one bigger drop.]
Now the Chinese room thought experiment is not about the computational model of consciousness—not directly! It was always phrased as a rebuttal of the Turing test in particular, and the computational model only indirectly after the Turing test falls. Note that the Turing test has no direct beef with the casual sufficiency axiom, which is why it went unstated originally. According to the Turing test, written text goes in, written text comes back out, a dialogue appears to happen to the outsider, this is sufficient to conclude conscious understanding of the language used, which confirms consciousness.
Searle’s objection is, “if it's really all about inputs and outputs and not how I get it done, then you've left out what for me feels like the most important part about understanding a language: understanding it is part of how I get it done!” Right? You understand English because you can phrase your ideas into it, you can mold it to suit you, and it can (when heard) so mold you—it’s not just because the words can come out of your mouth, triggered by other words that came in through your ears. The Turing test has always ever stated “don't worry what happens in the middle” and Searle is saying “but for me understanding is part of the details that are happening in the middle!”
So where does the computational theory come in? It comes in because Searle wants to make this argument rigorous! He says, “if your computational theory is true, then there is in fact another way that I can speak a language—words in, words out—where I don't understand a word of Chinese. I am Turing complete, I could memorize a program for speaking Chinese and happen to execute it flawlessly and at no point would my ideas get into Chinese, at no point would the things that I heard mold me. So the Turing test is crap!”
The prominence of the words “I, me” is what invokes the idea of causal sufficiency here, “I have the right inputs and outputs, but I don't understand.” The Turing test is not able to distinguish between multiple consciousnesses instantiated by the same hardware, if such is even possible. The inputs and outputs go into the same box, as far as Turing is concerned as long as only one conversation happens, there's only one person in there.
This does do a great job of defanging the Turing test, because all of the ways that you might weaken the concept to address this major limitation do make it sound completely tautological. “Yeah well something consciousnessy is happening in that box but I don't know what.” / “Okay then why are we even talking about it.” / “Because computers can speak!” / “Right, so we care that computers can speak because they can speak?” / “No, like, we gotta give them rights now, or some shit.” / “John Searle already has human rights, if he's memorized a program that lets him speak Chinese without understanding it, you're saying we need to give that program human rights?” / “Yeah!” / “So uh is it murder if Searle decides that running the program is no longer fun? Is he a slave to this program forever?” / “Uhhhh...”
It doesn't defang the computational model, not directly. But the computational model does imply that VMs exist. We use them every day! And that's all that the Chinese room is, it's running a VM inside of another computer, one consciousness carrying a separate consciousness inside of it, a willful sort of demonic possession. The only thing the Chinese room has to say about this, is that we don't use our language very well if it is true. Philosophers who believe in that will need to generate an alternative language that is able to distinguish between “I am doing it” and “I am sustaining a daemon who is doing it,” because for them that's a real difference, you might have a hundred consciousnesses in your head that you don't have direct access to. That is a necessary part of believing that consciousness is software, you don't know if you're in a VM inside your brain, you don't know if something you're doing is actually secretly a Brainfuck program instantiating another VM inside your consciousness, software embeds within software, that's a core feature of software.
But of course Searle thinks that that's kind of ridiculous because he thinks that it's obvious that consciousness is something that the squishy wetware of the brain does, and this forces him to believe in that causal sufficiency—“my brain is only sustaining one consciousness, namely me,”—which the computationalists cannot ever agree with because that's not how software works. Anyone who believes in that causal sufficiency, even if they don't have the same basis that Searle does for believing in it, also thinks that the computationalists are ridiculous.
But the point is, that's happening at a level way above the Chinese room argument, Chinese room baseball came and went, now this is a whole separate game being played at the same ballpark afterwards.
LLMs approach learning differently -- whatever reasoning they do possess is an emergent property that arises as they get better and better at language generation. In other words, unlike humans, LLMs learn to "speak" before they exhibit behaviors that look like logical reasoning. Humans do the opposite.
It would be both amusing and disturbing to see older Usenet, Slashdot, and Reddit conversations turned into rare and valuable resources.
Certainly, here's an exponential equation that can be challenging to solve for integer solutions:
2^x+3^y=7^z
This equation involves three variables, x, y, and z, and requires finding integer values for these variables that satisfy the equation. This type of equation is known as a Diophantine equation, and solving it can be quite challenging, especially for larger values of x, y, and z.
"
I mean, it's definitely not a Diophantine equation and solving it is definitely not challenging -- (2, 1, 1) happens to be an easy solution -- but I want to say it probably doesn't have any other solutions but I don't see a great way to prove it...
Indeed, but I'm not trying to with that comment, which is just about how my own seems to work on self-reflection. That said, I do have reason to doubt the accuracy of human introspection of our own thought processes, and therefore my own judgment in this may also be flawed.
That said, this all seems like a 'fallacy of composition'. Humans are not LLMs, so much should be obvious, but at the same time concluding that they are completely different just feels wrong. The mistakes LLMs make feel very similar to what humans do when they don't have the time to deeply think about a problem and just give you the best guess that pops into their head. Humans will have other systems on top that allow them deeper reasoning, but the language generation really doesn't feel all that different from what LLMs do.
That aside, humans interact with the world, they get instant feedback on what of their predictions is right or wrong. LLMs are stuck with just static training data that might simply not be enough to develop higher level reasoning skills.