News for hackers. Stuff that matters.
Re:Artificial intelligence is not conscious – Ted Chiang by Nevermark on Wednesday June 03, 2026 @07:37PM (#48391605)
> My intention is to highlight the fact that LLM conversations are cleverly disguised examples of sentence continuation

Regardless of bigger issues, this kind of statement reveals a deep misunderstanding.

Problem type does not limit problem complexity. Nor does problem type limit solution complexity or power.

If a machine has to learn to understand humans to complete text, then that is what it has to do. And there is no theoretical or practical basis for suggesting that this is somehow "faking" understanding, just because of the form of original data streaming in and out.

Neither problem type, nor input/output structure, limit internal representations.

Understanding is learned from patterns in the data, not the gross form of the data. Does the data require an understanding of something to complete the task? Then that understanding will be what is optimized.

To the degree they are limited, it is for other reasons. Resources such as computing, parameter number, lack of representative data, ... Which in the cases of SOTA models, we know are not limits. A conclusion verified by the models' actual abilities.

Re:Artificial intelligence is not conscious – Ted Chiang by cauch on Wednesday June 03, 2026 @08:10PM (#48391943)
I think, for me, the thing is that when you do basic ML, you discover that ML will very often find data pattern that fit the goal but does not correspond to a real mechanism.

So, I think there is a flaw in the logic of saying that human text have a pattern of "consciousness mechanism" and therefore LLM will learn "consciousness mechanism" in order to return sentence continuation that is convincing. There is probably tons of data pattern that LLM can learn from to be able to reproduce a sentence continuation that is convincing without having to learn the specific mechanism that is "conscious".

For me, one element that shows it is the case is the absence of world model (or "human-like" world model) despite the fact that the sentence continuation is convincing. If indeed the only way to produce sentence continuation convincingly would be by "simulating a brain", then it would not explain the first LLM from several years ago (before the extra layers of RLHF, ...). They were able to have quite convincing conversation on a lot of non-trivial aspect, and yet failed on some aspects that should have been basic for a system that would have been trained to work like a human brain. It shows that it is possible to "cleverly disguise examples of sentence continuation" without having to build elements that one expect on a conscious being.

Re:Artificial intelligence is not conscious – Ted Chiang by Nevermark on Wednesday June 03, 2026 @08:17PM (#48392002)
I didn't make the claim that a model can learn consciousness.

Understanding is not consciousness.

Their training is all about understanding. There is nothing in their architecture or training that credibly optimizes for rich self-awareness.

Given non-persistent experience, non-continuous operation, no ability to build up generalizations and aggregate experience of their own self-awareness over time, they seem to be structurally designed to not have consciousness.

This is a case where acting is very credible. Understanding of other's consciousness, in a functional and third party sense, isn't a substrate for personal experience.

In stark contrast, humans develop consciousness gradually over continuous time with persistent aggregation of experience. By the time we can recognize our own consciousness in the abstract, and reason about it, we have had it for some time.

Re:Artificial intelligence is not conscious – Ted Chiang by cauch on Wednesday June 03, 2026 @08:31PM (#48392127)
I use "consciousness" because it's the point of the original argument, but in fact, I think my whole comment still work well if you replace "consciousness" with "understanding".

My point is that the fact that AI can reproduce convincingly human sentence continuation does not imply that the AI has no choice but ending up using a mechanism that "understand" rather than just have learned data patterns that are very effective to fake human sentence continuation but are meaningless in term of understanding the concepts.

And I think that if indeed the only way for AI to reproduce convincingly human sentence continuation would be to end up in a configuration that uses the "understand" mechanism to do so, the behaviour of the first LLM would not show that they are so good at sounding human and yet so bad at failing basic "understanding" tests.

Re:Artificial intelligence is not conscious – Ted Chiang by Nevermark on Wednesday June 03, 2026 @09:40PM (#48392605)
> the fact that AI can reproduce convincingly human sentence continuation does not imply that the AI has no choice but ending up using a mechanism that "understand" rather than just have learned data patterns

Taken as an absolute without any addition context you are right.

But we are not talking about abstractions but specific successful models. The number of parameters models they have may seem large, but they are very small relative to the training data that they have to summarize. That cannot do it without discovering that patterns that make sense out of it.

And we can verify that. Simply discuss completely disparate topics, with some kind of intersection. Converge several highly unlikely topics, there are so many it would take billions of years to exhaust unlikely combinations.

If the model is only interpolating it will produce gibberish.

But that isn't what happens.

The fact that models can be near expert, and sometimes expert, across vast areas of human knowledge is a clue. If they don't understand that, then the question is, why do we think people understand things. Does having an answer mean a human understands something, or is their intuition and stream of conscious reasoning also not understanding? To be even handed about what we mean by understanding.

Re:Artificial intelligence is not conscious – Ted Chiang by cauch on Thursday June 04, 2026 @08:51AM (#48397955)
> That cannot do it without discovering that patterns that make sense out of it.

I don't think it's true at all, and I think we have indication that proves it is false.

We have "basic" LLM, the ones from 2023. They were producing _very convincing_ human text, and yet, they were too often failing basic tests that require understanding.

Now, we have more advanced models, but the counter-example of "basic" LLM demonstrates your assertion is incorrect: these model _did_ produce very convincing human text and yet did not make sense out of it.

But for the more advanced models, the problem is that they are "on top" of basic LLM. So, the first step is a training that build a mechanism that produce convincing text without understanding, and then, the "residuals" are fine-tuned. The result is very unlikely to add "understanding" to the model, because to do so, the whole system needs to deconstruct the basic LLM, to go back towards less efficient situations in order to rebuild almost from scratch. The fact that modern LLM are based on basic LLM means that the first step put the cursor in the bottom of the "basic LLM mechanism" valley, which is a local minimum. And any layer on top of it cannot "climb up" the slope of the valley, pass the ridge and fall into the next valley, even if this next valley has a lower minimum.

> The number of parameters models they have may seem large, but they are very small relative to the training data that they have to summarize.

That is demonstrably an incorrect logic jump. For example, CNN are able to distinguish between pictures of cats and pictures of dogs. The weights in these models are very small relative to the number of pixels they have been trained on. Yet, they distinguish cats and dogs by finding specific shapes in the pictures, without understanding what a 3-D cat and a 3-D dog is.

They have done that without discovering the typical human pattern that make sense of "cat" and "dog". And yet, the number of weights is very very small with respect to the number of pixel used in training.

> And we can verify that. Simply discuss completely disparate topics, ... > If the model is only interpolating it will produce gibberish.

What you are saying is that the model is not simplistic interpolation. But that is a straw man argument: people who say that LLM don't understand don't say LLM are equivalent to simple interpolation machine.

But the problem is that you can have very good predictions in novel situations without understanding.

For example, if you have 10 totally different situations that can be described with a Gaussian curve, and that I show you points for a new situations that cover the left side of a Gaussian curve. Then you will be able to guess that the right side of the curve, which is not an interpolation as it corresponds to situations you never saw, will behave like the rest of the Gaussian curve. And yet, in these 11 situations, I did not even say which real physical phenomenon I'm talking about. You haven't understood anything about these phenomenon, all you have done is guessed that a typical pattern that you have observed somewhere else is more likely to apply here too, without even having to understand anything about the reality of this situation.

And of course, this prediction is "a guess": maybe, for once, in this 11th situation, the curve will start as a Gaussian curve but will suddenly be different. But it happens that the reality is that in this 11th situation, the correct description is a Gaussian curve (because, due to the maths, Gaussian curves are really common). So, when you make your prediction, it looks like you understand the situation, it looks like you understood the physical mechanism that applied here. But it is not the case.

So, no, correctly doing such prediction does not demonstrate understanding.

> The fact that models can be near expert, and sometimes expert, across vast areas of human knowledge is a clue.

That is not at all sufficient. A Chinese room experiment will do that despite the system not understanding Chinese. A pocket calculator will be able to be expert in math computation.

> If they don't understand that, then the question is, why do we think people understand things.

That's the wrong question. The correct question is: we know people understand things, and we see AI behaving similarly to people in some aspect, but is this behaviour _requires_ understanding, or can we reproduce this behaviour without needing to understand?

The fact that "basic" LLM were able to reproduce very convincing text that look like they understood X and yet were demonstrably showing lack of understanding of X demonstrates that we cannot just jump to the conclusion that just because it looks the same, the only possibility is that the core mechanism is identical.

Re:Artificial intelligence is not conscious – Ted Chiang by lstodd on Thursday June 04, 2026 @04:15PM (#48404026)
Thank you for writing this out.
Re:Artificial intelligence is not conscious – Ted Chiang by hatthew on Thursday June 04, 2026 @05:56PM (#48405146)
I think most debates about LLMs understanding boil down to different definitions of the word "understand." For example, with the definition of "understand" that I typically use in my daily life, I would argue that in the chinese room, the system as a whole "understands" chinese.
Re:Artificial intelligence is not conscious – Ted Chiang by cauch on Thursday June 04, 2026 @06:46PM (#48405707)
Fair enough, but then, a pocket calculator also understands math, and a pocket translator also understands language. And a wikipedia page that inform you about radioactivity understands nuclear physics. Some will maybe say it is the case, but if we talk about the LLM capabilities as a novelty, then it implies that we are talking about something else, because otherwise, it is not novel at all and it does not make sense to pretend it is.
Re:Artificial intelligence is not conscious – Ted Chiang by wan23 on Thursday June 04, 2026 @11:06PM (#48407461)
Does a pocket calculator not understand arithmetic? What part of a fourth grader's understanding is missing?
Re:Artificial intelligence is not conscious – Ted Chiang by cauch on Friday June 05, 2026 @09:54AM (#48412614)
Not sure what you mean.

I'm happy both ways:

Either you say that a pocket calculator understands arithmetic, and that LLM understand language, which is something trivial. If a pocket calculator understands arithmetic, than previous substitutes to calculators, such as an abacus, do too. In this case, a word dictionary also understand language. And it is basically what Chiang's article says: the LLM don't understand language more than a word dictionary does. If you disagree with Chiang, it looks like you do only because you don't understand what he is saying, or somehow are not mature enough to realise that Chiang may use a different definition of "understanding" than yours in a fully legitimate way, like everyone is always doing when talking about plenty of subject.

Or you pretend that a pocket calculator understanding of arithmetic is somehow different than the one of an abacus or other obviously inanimate object who are obviously not thinking.

Re:Artificial intelligence is not conscious – Ted Chiang by hatthew on Thursday June 04, 2026 @11:57PM (#48407747)
I'd say that broadly speaking, a system understands things when it can interact with them "correctly". I agree a pocket calculator understands math, but I'd say a pocket translator understands grammar, not language as a whole. A wikipedia page does not interact with anything, so I'm not interested in pushing the definition that far. However, if the wikipedia page were to make recommendations for nuclear safety based on some context it receives as input (say via an integrated LLM), I'd be happy to argue that it understands [that part of] nuclear physics.

I don't think that LLMs as black boxes are fundamentally novel, I just think that their internal design is novel, and their generality and ability to give correct responses to complex topics is far beyond anything previously. For example I would argue that wolfram alpha has a poor understanding of language and a very good understanding of math. I would argue that LLMs have an excellent understanding of language and a mediocre understanding of math, but are able to temporarily increase their understanding of math through document retrieval and "thinking" (or whatever you want to call the process of iteratively generating tokens that build on each other to result in a final response).

Re:Artificial intelligence is not conscious – Ted Chiang by cauch on Friday June 05, 2026 @09:59AM (#48412685)
Well, then you basically agree with Chiang's article. Just that Chiang as a clever usage of the word "understanding" than you (more clever because more nuanced: 1) I doubt that "people on the street" will agree that obviously "brainless" objects, like a pocket calculator or an interactive wikipedia page will understands anything, 2) Chiang is not stumbling on words: he explained his case that makes clear what he means, and it is to the interlocutor to adopt his vocabulary (because it is very legitimate here) rather than start saying "hm, no, I disagree, because for me, 'conscious' means 'print something on the screen', so LLMs are conscious". That is just missing the point)
Re:Artificial intelligence is not conscious – Ted Chiang by Nevermark on Thursday June 04, 2026 @02:18AM (#48394715)
It turns out that the optimal way to highly compress complex information is to understand it.

Sometimes, a problem being hard means you only get bad solutions, or increasingly accurate ones.

The planet isn't big enough for the proverbial interpolative stochastic parrot, over the training set of global human communication.

Re:Artificial intelligence is not conscious – Ted Chiang by cauch on Thursday June 04, 2026 @08:59AM (#48398035)
Two problems with that.

Firstly, how do you know that the optimal way to highly compress complex information is to understand it? You think it is obvious because you are very familiar with "understanding" as a way to summarise complex information. But there can be billions of different ways, outside of human imagination, that is as good or even better.

But secondly, LLM don't find the optimal way, they find the local minimum. Everyone who worked with NN knows that they are prone to come up with spurious pattern, incorrect correlations and bad workaround to guess the correct answer. You regularly need to nudge the NN by creating specifically engineered features to avoid them to fall into the first local minimum.

When it comes to LLM, it is extremely complicated to control to see if the LLM has triggered on a misleading pattern that, by chance, links two "tokens" together, or on a real concept that indeed links two "tokens" together. Basic probability implies that there are probably tons of "fake patterns" engraved into the weight during the LLM training, "fake patterns" that should not exist if there was any kind of "understanding" of the abstract mechanism that links these tokens.

Re:Artificial intelligence is not conscious – Ted Chiang by Nevermark on Saturday June 06, 2026 @12:25AM (#48421372)
> Firstly, how do you know that the optimal way to highly compress complex information is to understand it?

What is your non-performance baseline for "Understanding"? We don't have such a measure for humans.

Understanding is the behavioral ability demonstrated by learning to model something complex well. Beyond mappings, associations, interpolations.

Models clearly do. Mix up the most unlikely combination of non-trivial subjects, and they response sensibly. Those are not averaged, interpolated by any order, or even combinatorially interactions.

There is a reason those kinds of encodings, mappings, associations, interpolations, statistics / stochastics, all failed miserably for decades. Still fail. It took topological transforms, reminiscent of how we compute (dendrite-soma-axon, tensor-sum-nonlinear), and then they lept several orders of magnitude ahead of any alternative.

The problem with models composed of relationships of lower order than the phenomena they are trying to model, is they require combinatorially more parameters to model anything complex.

For simple problems, poor models fail gracefully. For complex problems, poor models just fail.

Re:Artificial intelligence is not conscious – Ted Chiang by cauch on Saturday June 06, 2026 @06:27PM (#48429646)
> Models clearly do. Mix up the most unlikely combination of non-trivial subjects, and they response sensibly. Those are not averaged, interpolated by any order, or even combinatorially interactions.

How do you even know it is the case?

How do you know the output is not the result of combinatorial interactions?

How do you even know that the "sensible" response on unlikely combination is not the result of a simple recipe that "make the response sounds sensible"? Either you, yourself, have some expertise on the subject, and therefore the combination does probably exist in the AI training data, or you don't and you have no idea if the response is sensible or is the usual smooth talk that everyone could come up spending 2 or 3 hours googling on the subject and crafting something sensible.

Worse, you are saying that the model "understand", which means that it discovers the underlying mechanism that drive the output. This "understanding" is a set of equation that link different concept, that explain how one concept affects another concept. So, it is "combinatorial interaction". Not a simple linear one, but guess what, LLM are designed to introduce non-linearity.

Even when AI are able to find new solution of math problem, the result is, like when done by humans, by using existing basic tools to build more complicated ones.

> It took topological transforms, reminiscent of how we compute (dendrite-soma-axon, tensor-sum-nonlinear), and then they lept several orders of magnitude ahead of any alternative.

And yet, the LLM elements that are "similar" or "analogue" to how the human brain works are very small. The human brain has thoughts "flowing", while LLM can only work "by step". The human brain is able to learn on a very reduced dataset, while LLM need more data that a human will ever be able to analyse, even less store. The human brain has "memory" and "context" intrinsically intertwined with how it works, while you can decouple these from the LLM. ...

Finally, here is a good contradiction of having you in one side saying that AI is mimicking the human brain and it is why it works well and on the other hand saying that AI will find the lowest minimum and that this minimum is "understanding how the phenomenom works" rather than "repeating by hearth what it was told during training".

As a human, when you mentally compute 6 times 7, what do you do? Do you do: "6 follows 5, which follows 4, which follows 3, ... and 7 follows 6, which follows 5, ... so we have (1 + 1 + 1 + 1 + 1 + 1) times (1 + 1 + 1 + 1 + 1 + 1 + 1), which is 1 + 1 + 1 + 1 + ..."? I guess you probably don't, you just remember the most helpful element you remember by heart. For example, you remember by hearth that 6x7 is 42. Or you remember that 3x7 is 21, and therefore 6x7 is the double, 42. Or you remember that 7x7 is 49, and therefore 6x7 is 42. Or even have a "feeling" from a mixture of all these (6x7 is somewhere around 40 because 5x7 feels like being around 30 and 7x7 feels like being around 50, and if I think of number in the 40 that "feels" like they are from the 7-multiple-table, I remember 42).

Same thing when a human does 324x42: the majority of humans will decompose it in "simpler" multiplication that they remember by hearth and, and only then, they will combine them. It is a good example of how the brain optimise: by balancing the trade-off of "using memory" and "using understanding": basic operations use memory, but of course it is inefficient to use memory for all numbers, in which case it will use a combination of both.

The way human do basic math operation is not purely by "understanding" arithmetic, it is by relying on what they remember from their training. At the same time, humans know how arithmetic works, and they will use it when relevant. Yet, the human brains prefer to rely on some "learnt by hearth" elements. This is in contradiction with your assertion that optimisation will always lead to "understanding" and that human brains is optimizing the same way AIs do.

This is only one example with numbers, but of course it works with plenty of other things. This is also exactly why humans get "the wrong idea" on plenty of phenomenon, that are then described as "counter-intuitive".

The reason "by hearth" is part of a good strategy rather than "purely understanding" is because there is a trade-off between "memory" and "compute", in both the human brain and AI: it is easier (and therefore a stronger attractor during the optimisation of the process of "getting the correct answer") to do the faster operation "retrieve from memory" than to do the slower operation "retrieve the theory from memory, compute the first step, store it in the short term memory, compute the second step, store it in the short term memory, compute the final answer by adding the first step answer and the second step answer".

Re:Artificial intelligence is not conscious – Ted Chiang by Nevermark on Sunday June 07, 2026 @12:01AM (#48431652)
> How do you know the output is not the result of combinatorial interactions?

Because combinatorial isn't deeply nonlinear in a compact way. It can only combine deep nonlinearities by using exponentially more parameters. Handling each different nonlinear combination requires more parameters.

You are right, if models were learning combinatorial representations, those would not be understanding in the way your or I think of it.

But, all those approaches failed. Dramatically despite decades of work.

Because we don't have enough matter in the galaxy for that number of combinatorially exploding parameters.

Ask a model to: Write up a Supreme Court pleading for the rights of whales based on all that is known about them scientifically, recent whale language developments, and any applicable human rights law, given the relevant Supreme Court is in a parallel universe in Zion of the Matrix, being pleaded by Keanu Reeves, the actor not the character, and written in Dr. Seuss prose, except with as long of sentences as are needed to carry the real technicalities of a suitable filing. And include the assumptions of a back history of whales which have sequestered themselves into a deep hidden underground ocean, where they have been safe until recent excursions by humans which have harmed them. Be specific creating a real history behind those events, with details that are highly relevant to the motivation, reasoning and requests of the pleading. Avoids words with q where possible.

If you get anything readable at all, that isn't a combinatorial construction.

SOTA models have a lot of parameters. But nothing like what combinatorially composed "human-like intelligence" requires.

Re:Artificial intelligence is not conscious – Ted Chiang by cauch on Sunday June 07, 2026 @07:02AM (#48433665)
> REASON 1

This just means "simpler representations are not enough", not "good representations cannot be complex combinatorial combinations" (complex enough that it is very different to see them for a human).

> REASON 2

Are you saying that I believe that the only way to get human-like text is by doing a near-infinite one-to-one mapping? This is obviously not the case.

You can do, for example, a GAM time-series forecast. This can have a relatively low number of weight, and still return very sensible prediction, and yet not capture the real understanding of the phenomenon they will predict. For example, it does not understand causality, just correlation.

> REASON 3

That is like saying "I've built and algorithm that is able to do 10 + 27, but there is an infinite list of number, so it is impossible for this algorithm to do 23113454453 + 1233253245". That is not true, you just decompose into (53+45), (44+32), ... and add rules to combine these elements together.

It is what is happening with AI: there is enough data to get "some pattern" in the language. Just the patterns, not the understanding of the language itself. And this pattern can be reproduced in plenty of different places.

> REASON 4

This argument is contradicted by "basic LLM" or even simpler model that are performing surprisingly well. Less than SOTA, but if your argument is true, CNN or ARIMAX could never provide better than a coin toss.

> REASON 5

Your example is a good place where the AI will _combine_ patterns learnt from different place. It will pick characteristics of each of your scenarios, and mix them together. The result will look realistic, but it is still applying learnt pattern together.

Also, you did not answered about my human arithmetic, and all your reasons are contradicted by my example there. Humans DO maths partially because they "learnt by hearth" some pattern rather than apply the understanding of fundamental arithmetic. If "answering very well based on pattern" was not a good strategy, or was necessitating infinite weights, or was making it impossible to use these patterns in novel situation, how do you explain that human can even do that themselves? As soon as we admit that humans do "some pattern some times", than we have to admit that there is a continuous spectrum and admit that it allows output that looks realistic being the result of pattern rather than understanding.

By the way, I just saw a new article reaching HN: https://news.ycombinator.com/item?id=48410427 , and it is indeed explaining similar things, and illustrates that the best way for SOTA to deal with arithmetic is by "not understanding it". And yet, when you use one of those SOTA, you would be able to argue each one of your "REASON" to pretend that the model did understood arithmetic.

Re:Artificial intelligence is not conscious – Ted Chiang by Nevermark on Sunday June 07, 2026 @08:18AM (#48434084)
I am not sure what you mean by complex combinatorial. If we are talking about combinatorial, its combinatorial. N can be very large, but it is going to scale like combinatorial, not something else.

I just started out with mapping to be systematic. Mapping is ground zero, then interpolation i.e. any smooth fitting function or basis, then combinatorial where different bases are recognized and then project relative to their relevance to a new input.

Each of those increase modeling efficiency and power, but even combinatorial doesn't scale to problems like language.

I may be doing a poor job communicating. A formal breakdown of the scaling issues with lower order, but scaled to make up for it, modeling would be a great paper.

To prove me wrong (as a thought experiment), choose a lower order model, any kind you can imagine that would qualify as modeling without understanding. Demonstrate it can do anything close. That it could possibly scale to the human corpus with just a trillion parameters.

If it the number of parameters goes up far too fast, then that can't be the way deep learning solves the problem with a trillion, or a few billion, either.

And consider the other side. We have no idea how our own brains are lifting up what is relevant vs. what is not. We are used to it happening. We call it "understanding". But we don't know how it works, how we work. Despite experiencing it.

What we do know, because combinatorial is too resource intensive, is we are not just combinatorial either.

Re:Artificial intelligence is not conscious – Ted Chiang by cauch on Sunday June 07, 2026 @12:05PM (#48436147)
> I am not sure what you mean by complex combinatorial. If we are talking about combinatorial, its combinatorial. N can be very large, but it is going to scale like combinatorial, not something else.

The way a LLM works is by creating a space of N dimensions, N being the number of token. This space contains all the possible combinations. The LLM will find the best combination, but will not scan the whole space. To find the best combination, it will minimize the loss function, which is low when the output corresponds to the target. By doing so, it will not explore the combination that "goes in the wrong direction", and therefore it is not true to say that increasing the space as a scale S corresponds to increasing the difficulty of running the model by a scale S.

Because of that, while the combination space scales like combinatorial, the model does not. A model with 2 weights (or rather tokens, but the number of weights should be at least the number of tokens) corresponds to 4 combinations (AA, AB, BA, BB can indeed be described by 2 binary weights of value "A" or "B"). A model with 3 weights corresponds to 9 combinations. A model with 4 weights corresponds to 16 combinations. ... A model with N weights corresponds to N to the power N combinations. The number of combination increases a lot, and yet the number of weights increase linearly.

In SOTA, we have billions of weights. That is a model that contains a very very very very big number of combinations, something so big that it is difficult to understand for a human. It will not try all of these combination one by one, the gradient descend method will help it finding the best combination without having to do so.

So, yes, SOTA are finding "the best combination" amongst an impressively huge number of combinations, yet without having to "scale like combinatorial".

> To prove me wrong (as a thought experiment), choose a lower order model, any kind you can imagine that would qualify as modeling without understanding. Demonstrate it can do anything close. That it could possibly scale to the human corpus with just a trillion parameters.

Yes. Easy. A SOTA LLM does that. It is a modeling without understanding. It does not understand, it finds the best patterns. And when you put it in a new situation, it uses these patterns to create a new text, without truly understanding the content of the text. And if you ask an additional question, it will use the previous text as context, and create a new text that, as it has been trained to, will be consistent with the output that has been given.

Your assertion "you can prove me wrong" is a circular reasoning: you start saying "if a model can do a text that looks realistic to me, then it means it has understanding. To prove me wrong, give me a text that looks realistic to me and has no understanding". Well, I cannot do that, because for you, if it looks realistic, it has to have understanding.

> If it the number of parameters goes up far too fast, then that can't be the way deep learning solves the problem with a trillion, or a few billion, either.

The combination space grows as N to the power N. So, a trillion parameters is not "just 1000 times bigger" than a billion parameters, but more than 1000 to the power of one billion bigger (the exact value is often even bigger than that). Do you realise the size of the combination space? That is 1 followed by 3 times one billion zeroes.

> What we do know, because combinatorial is too resource intensive, is we are not just combinatorial either.

I think you don't understand how LLM works: the find the best combinations in a incredibly huge parameter space, but don't need to explore the whole space, just the 1-dimension manifold that is the curve that follow the gradient descend within this huge combination space.

There are plenty of clues that SOTA don't "understand". For example, did you notice that SOTA happens to understand what human understand, and don't understand what human don't understand. If indeed the way SOTA works would be by "discovering the true mechanism", it means that it would discover with equal probability mechanisms that humans have already noticed and mechanisms that humans have not already noticed yet. For example, humans know that the Standard Model of particle physics is incomplete, and there are plenty of texts and books about that that the SOTA learnt about. Yet, SOTA did not "understood" the underlying mechanism that explain particle physics. It does not really know what an electron is by "making sense of what this object does", it only knows it as "a language word that can be used in some context in a specific way".

And, sure, SOTA is helping with new discoveries, but the way it does it is by using "reasoning" approach. If indeed SOTA creates its own understanding when learning the human language, then it should have the new discovery after the learning, without using any "reasoning" approach, because it would be something that it has already understood.

Re:Artificial intelligence is not conscious – Ted Chiang by Nevermark on Monday June 08, 2026 @01:37AM (#48441667)
> Well, I cannot do that, because for you, if it looks realistic, it has to have understanding.

Yes, if it consistently produces good output for highly varied stimuli that can be intentionally picked to have been unlikely to ever had obvious representation in the training set, then yes it understands.

I think we are talking past each other a bit.

A series of increasingly challenging datasets, used to capture scaling efficiencies, would ground our discussion.

But the level of performance for models is simply too good vs. the number of parameters to be doing anything trivial.

Deep learning models do something combinatorial models do not. The linear tensor + non-linear transforms do two special things:

1. The tensor itself just projects a linear space into higher dimensions, but its still the same information space. Project a 2D surface into higher dimensions linearly, and there can be more parameters, but it is not more information, since there is an expansion of linear dependence to match.

2a. But then the nonlinear both (a) thresholds, squashes or otherwise alters the linear results, in a way that removes linear dependencies, increasing the useful dimensionality of the representation.

2b. And the squashing also allows dimensions to be folded down.

So by both expanding and flattening representational dimensions, deep learning models are able to model higher-order relationship directly, that any less expressive modeling would require cobbling together many patches of fitting.

Another way to put this, is deep learning models are able to learn higher-order relationships directly, not be memorizing and interpolating across learned points or regions.

So a dramatically greater ability to "understand" is why deep learning models are so much better. They are not doing simple combinatorial fitting.

"Understanding" or not, combinatorial relationships are the low bar for deep learning models, they are inherently great a learning much higher-order relationships.

I am falling asleep at this point. I feel like we need a blackboard and a computer. You are saying a lot of things that make me think, and make sense to me.

Re:Artificial intelligence is not conscious – Ted Chiang by cauch on Monday June 08, 2026 @09:15AM (#48444953)
Yes, this conversation is useless.

You keep saying "what I observe with GenAI can only be the result of 'understanding'" without providing any proofs at all. Just few beliefs.

You just say "look at this behavior, that's the proof". I truly don't think it is: nothing proves that this behavior requires 'understanding'. And nothing you provided helps: all you provided are impressive behaviors and then the unsubstantiated conclusions "and this behavior can only be done with understanding".

At the same time, there are too much clues showing that such behavior does not require understanding, even if it _looks_ incredibly clever:

1. GenAI does not understand (after the training phase) things that humans don't understand. If GenAI had the capacity of building an understanding during training, then there is no reason this understand will coincide with human understanding.

2. Optimisation does not always lead to "understanding". Human brains choose to optimise "learning multiplication table by heart" rather than building a pocket calculator inside the neurons.

3. Human brains, that have "understanding", are working fundamentally differently from GenAI (flow of thoughts, intrinsically intertwined memory and compute, optimised for world-model treatment rather than token treatment, ...). It is an unsubstantiated jump to simply conclude AI has "understanding", while it can be the result of fundamental differences.

4. "Basic" LLM are surprisingly good at creating convincing sentence and yet there are situations where it is blatantly clear they did not understood anything. More advanced SOTA are based of refinement of "basic LLM", and therefore the "sentence construction that is done without understanding" is still used, and impair the SOTA model to build a full understanding.

> Another way to put this, is deep learning models are able to learn higher-order relationships directly, not be memorizing and interpolating across learned points or regions.

It's exactly what I'm saying: deep learning models are very good at learning complex relationships. Such as "I don't know what 'Paris' is, I don't have any understand of what a city is in reality, but when the token Paris is associated with these other tokens in this complex order, even if I never saw it before, I have learnt the complex relationships and therefore I'm able to build a series of token".

They are very good at learning complex relationship that allows them to choose the correct combination even if they did not "understand" the content of the correct combination.

I understand that it is impressive: those relationships are very complex and very numerous (there are billions of them). It is easier to do anthropomorphism and conclude that the AI has "understood".

But again, the main problem is that you just pretend, without any proof, "no, I cannot believe that, I refuse to believe that".

(and, by the way, I personally think that AI (SOTA but also even "basic LLM") do have 'rules' that correspond to some kind of understanding of basic mechanism. I think they have basic "world models". But these world models are optimised "to write text" rather than to "understand the world", and therefore the large majority of AI output is just not-understood token chains)

Re:Artificial intelligence is not conscious – Ted Chiang by Nevermark on Monday June 08, 2026 @05:33PM (#48452460)
Apologies. Your pushback (frustration and patience) has helped me crystalize my view, thank you.

1. Define understanding.

My definition isn't vague: "a compact representation enabled because that representation's topology closely matches the topology of the relationships being modeling."

Understanding = Scope and Suitability of Behavior / # Parameters.

Useful property: This definition applies across all scales: Scientists and mathematicians increase our understanding, every time patchworks of relationships get replaced with a simpler underlying insight.

Another useful property: It distinguishes between better understanding and having more facts. Facts improve performance but do not (non-trivially) decrease parameters.

What is your definition? In measurable terms?

2. You keep avoiding a basic aspect of modeling:

Higher compactness is achieved by higher representation correspondence between a model and the modeled.

Yes, lower level representations can work. Even well, without good "understanding". But not as compactly. And as problem complexity grows, the relative difference in parameter budgets for high-correspondence and low-correspondence representations explode.

This is not a subtle effect.

The hallmark of lower-level fitting is the far greater number of parameters required.

Dead simple example: Piece-wise linear vs. polynomial fitting of Bezier curves. Accuracy / parameter is far greater for the latter, because the representation matches the relationships being modeled.

That is an intentionally trivial example, but the same relationship holds for any problem.

You keep avoiding that.

3. Today's LLM models are very compact compared to humans.

Compressing the substance of a corpus of global human writing into less than 1% of a single human's parameter space is compact.

Humans have 100–200 trillion, some people think 500 trillion, synapses.

How do you argue that behavior scope and suitability / parameters is not remarkable, when it is remarkable compared to any specific human you could point to?

No human can converse reasonably across the scope of global communication. But these models can. For <1% of a human's parameter budget.

4. Finally, based on your clear definition, how do you argue that humans understand but models do not? Saying we are different is a copout. Defining understanding as us vs. other is both circular and unenlightening. And ignores the real progress models are clearly making relative to humans.

Re:Artificial intelligence is not conscious – Ted Chiang by Nevermark on Tuesday June 09, 2026 @12:03AM (#48456227)
Is that more coherent?
Re:Artificial intelligence is not conscious – Ted Chiang by cauch on Tuesday June 09, 2026 @07:19AM (#48459575)
1. Okay with your definition.

My point is that you can have the same result with a representation that "closely matches the topology of the relationships being modelled". For example, a representation that "allows relationships between tokens but yet does not care about the meaning or concept not useful to form convincing sentences".

And therefore, it means that you can have convincing text without needing a "representation's topology closely matching the topology of the relationships being modelled", and therefore, according to your own definition: no understanding.

2. It is not true I'm avoiding that. I have answered very clearly.

1) GenAI are not trained to get the higher representation of the world, but to get the best convincing sentence generation. This does not require a full world understanding. Worse, once a convincing sentence generation is reached, there is no gain by getting a better world understanding: the training mechanism that pushes into the correct direction stops and therefore it can go into any direction at all.

2) High compactness does not equal best solution. Even humans don't used "high compactness" when doing basic arithmetic, but use "by heart multiplication table". Being compact is useless if it comes with high complexity each time you need to recompute the output.

3) Very very good approximation can reach higher compactness anyway. Your Bezier curves is a good example: real physical phenomenons are almost never the result of a Bezier equation. A Bezier curve did not understood the phenomenon. When it comes to GenAI, it can "fit" the reality with very close precision with several representations, but the majority of the representation corresponds to an incorrect "understanding" of the reality.

Another example: if I throw a ball in the air, the motion will be at first order a quadratic equation, plus correction due to friction, wind, ... If I just "train" something for "throw a ball", this system may fit a quadratic function plus corrections, but they will achieve the same result with Bezier curves, or Fourier series, or additive Gaussian, or ... But the "understanding" is that the ball is influenced by gravity, which leads to a quadratic equation. The system does not understand that. It has no reason to understand that. And it has no reason to prefer a quadratic equation fit rather than a Bezier fit, on the contrary, the Bezier fit will be more realistic (as the quadratic equation is just the first order approximation).

If you want to understand a paper plane trajectory, it is a complex system, and you probably need plenty of parameters to describe the gravity, the wind at each position and each time, the shape of the plane at each time, ... But you can describe the trajectory with just few parameters using a Bezier curve. Train on plenty of paper plane trajectories, and you will have a system that can give you a very realistic paper plane trajectory based on Bezier curve. And yet, your system has no understanding of the paper plane trajectory: it does not know what are the mechanisms that make the paper plane goes up or down. It just creates a realistic trajectory without knowing why this trajectory is realistic, just that this trajectory makes sense based on the other trajectories it has seen.

3. This argument seems to go against your thesis. You are saying that humans, who "understand" + are not even able to have as much conversation as LLM, have way too much neurons. What are these neurons even for then? You are explaining that LLM are just "something different", a reduced mini-version of a brain, and yet you are also saying that they are able to do the complex things the brain do.

Another way of seeing it, is that LLM are "dropping" things that they don't need to create convincing sentences, such as "understanding the token". They just "get the Bezier curve fit of the relationship" instead of understanding the real mechanisms and concepts.

It's like your Bezier curve example: a system that just creates a realistic paper plane trajectory based on "typical Bezier curve observed during training" will need way less "neurons" than a system that needs to understand the whole aerodynamism of the paper plane.

4. I argue this the same way I say that a system that describe a paper plane trajectory based on best Bezier curves did not understood the mechanism behind how a paper plane trajectory works. I am not saying "I define 'understanding' as what humans do", I am saying that creating convincing sentences does not require understanding, the same way that generating realistic paper plane trajectories does not require understand gravity, Navier-Stokes equations and Brownian motions.

The Bezier curve paper plane trajectory predictor system I have mention, do you think it has understanding of gravity? of Navier-Sotkes? of Brownian motions?

No, it has not. You can open this system. It just has Bezier curve for plenty of examples, and thanks to that, it knows that one trajectory is realistic and another is unrealistic. And at some point, it is also able to give realistic trajectories in brand new situations it has never trained on.

Re:Artificial intelligence is not conscious – Ted Chiang by Nevermark on Tuesday June 09, 2026 @03:34PM (#48466483)
I am going to tune the expression form of my definition to:

Understanding = Novel Scope * Suitability / Parameter Count.

> My point is that you can have the same result with a representation that "closely matches the topology of the relationships being modelled". For example, a representation that "allows relationships between tokens but yet does not care about the meaning or concept not useful to form convincing sentences".

You are absolutely right, that lack of internal representation-reality correspondence does not rule out real/convincing performance.

> GenAI are not trained to get the higher representation of the world, but to get the best convincing sentence generation.

This is true of all learning. And it will always be the nature of learning.

Which is why performance is always (should be) measured on novel input.

> High compactness does not equal best solution. Even humans don't used "high compactness" when doing basic arithmetic, but use "by heart multiplication table".

This is a really good point!

It brings up the two useful modes of human representation:

(1) The brain's slow mode is very good at handling deeper and deeper layers of representation. When thinking about arithmetic or more complex math analytically, our understanding does follow a path of increasingly deeper representations. And we are very good at applying these deeper understandings.

(2) Then, our fast mode creates shallow representations of things we do frequently.

I would look at this as (1) reflecting scalable understanding (2) reflecting very limited understanding, but scalable speed.

And we often use both modes together.

I would argue that the understanding is primarily in the slow mode. That the fast mode, is the non-understanding but appropriate response mode. And that it operates with a much reduced scope of appropriate response, but a high percentage of applicability. Meaning, most of the time we don't need to use deep understanding we just need fast appropriate response.

But how to compare the two in scopes where they are equally accurate?

I think "high understanding" representations are those very flexible to being used in ways quite different from how they were learned.

Our slow mode does this very well. Our fast mode not so well, but to the degree it generalizes well to novel situations, that would be an increase in understanding.

Our fast system does generalize, but I would argue that at some point it fails, where our slower deeper representations provide the means of analyzing a situation. So it clearly "understands" better.

It is interesting how quickly understanding from our analytical side translates into operation on our fast side. Clearly, our fast side has very efficient access to new "patterns" that our slow side constructs.

> If you want to understand a paper plane trajectory, it is a complex system, and you probably need plenty of parameters to describe the gravity, the wind at each position and each time, the shape of the plane at each time, ... But you can describe the trajectory with just few parameters using a Bezier curve.

I love this example. It does contrast very different kinds of understanding.

(1) Understanding the fundamental reality in which paper planes exist,

(2) Vs. understanding how paper planes behave.

I think my expression works well here, as long as we take "scope" seriously.

Understanding = Novel Scope * Suitability / Parameter Count.

For paper planes as a hobby, a smaller neuron/parameter budget is achieved by learning the emergent laws of paper planes, not their underlying physics. And understanding paper planes is achieved with this smaller budget.

For understanding paper plane dynamics at a design level, a smaller neuron/parameter budget is achieved by learning the underlying physics of aerodynamics at an intuitive level.

For understanding paper plane dynamics at a world class competition level, a smaller neuron/parameter budget is achieved by learning the underlying physics of aerodynamics at an analytical level.

So these would be three different "understandings", each with their own scope and area of appropriate response to novel situations.

Point taken: The most fundamental correspondence isn't the point of a lot of understanding.

You are right, and my equation works, as long "scope" is interpreted to mean appropriate level of interest, not area of fundamental physics involved. Great point.

Does that get us on the same page? Closer?

> I am saying that creating convincing sentences does not require understanding

As problem complexity goes up, there really is an explosive difference between appropriate response via "familiarity" or lower-level fit, vs. higher level fit, for the same number of parameters.

And it is also a dramatically bigger challenge for lower-level fits to respond well to novel stimuli, given the same number of parameters.

The reason is, is that complex problems operate in higher dimensional spaces, and relationships in higher dimensional spaces have exponentially more complexity for any level of representation. Exponentially.

Linear fits of a 2D bezier are inefficient but work. Linear fits for a 100 dimensional bezier, which isn't very many dimensions from a data standpoint, become ludicrously expensive in parameters.

The dimensionality of human communication is probably the most complex problem ever tackled systematically.

I am trying to think of a way to capture this more concretely. I.e. a way to draw a line in this conversation that stands up on its own. All I can point to, is the complete failure of any lower-level fit when done directly, to acheive a trillionth of a trillionth of trillions of the flexibility that SOTA models demonstrate. The extreme dimensionality of input that LLMs respond to, makes my "trillionths" literal in this case. And we do get a concrete measure of the dimensionality within their capacity, as context windows give us live demonstrations of this.

Note that language is literally highly compressed information, with pervasive non-local interactions. The enormous dimensionality is compounded by dense reactivity, pervasive discontinuities. No other informational artifact compares to language complexity.

When I say that this is a case where either real relationships are learned or the model fails, it is because the number of parameters for a lower-level fit really are beyond imagining.

You can't point to any lower-level fit, where the lower-level fit is basic to the fitting algorithm, that ever achieved even a tiny-grammar tiny-subject-scope toy of a toy version, to what LLMs are doing. Nor can I, despite following progress for decades. Nobody can. The original successes of the first LLMs, modest as they appear now, were completely unprecedented.

There just are not enough parameters, by many orders of magnitude, to do language justice over a context window, and respond sensibly to intentionally novel conversations, without identifying the actual relationships behind it.

So that would be my challenge to you. To identify any verifiable lower-level fit that even approximates LLM behavior at the tiniest of toy levels. Verifiable fits at any given level are easy to do, just train a model where the basis is restricted to that kind of fit.

Otherwise, I can agree that understanding is a continuous property, and that how well something understands something, without strict benchmarking by well thought out benchmarks, involves intuition and judgement. So there can be legitimate differences in how we perceive model understanding, in the absence of direct measures.

Any more thoughts? I have understood both myself and your points better as we went along.

Re:Artificial intelligence is not conscious – Ted Chiang by cauch on Tuesday June 09, 2026 @08:23PM (#48469619)
Ok, I've stopped reading half way because it is useless. Your reflection does not bring anything concrete, it is just you trying to play on semantic and loose the point.

For example, you just moved the definition problem to "novel". Do you even realise that?

You are claiming that there is an understanding because the model is able to do something in a novel situation where only deeper understanding of the situation will allow it to perform that well. The big problem is that you have no idea if this situation is "naturally easy to reach" or not.

For example, a system that is fitted on the electrostatics Coulomb's law will build, internally, a set of equations to generate realistic predictions. And then you take this system and put it in a totally novel situation: classical gravitational problems. Well, this system will be able to generate realistic predictions there too, because Newton's law works with equations that have the same form.

When you are discussing with a LLM in a "novel" subject, how do you know the LLM cannot directly use the fit and complex equations that it has created for the "non-novel" situations it has been trained on? For example, the LLM has been trained on "Pride and Prejudice and Zombies" and tons of other mash-ups. Even if asking a story of Keanu Reeves and the Supreme Court looks "novel" to you, it does not mean the generated text was not in fact super easy to generate based on the patterns that the LLM has seen in tons of examples.

Honestly, this whole conversation just convinced me that too many people who claim "GenAI does understand" are way above their head on the subject. If you want to continue talking, just talk to a LLM. Plenty of people have done so and convinced themselves they were geniuses when in fact they were not at all. Yet another example that LLM has no understanding, as it is very very often failing to distinguish between correct ideas and "things that look correct but that someone with real understanding will not encourage".

Re:Artificial intelligence is not conscious – Ted Chiang by Nevermark on Wednesday June 10, 2026 @01:26PM (#48479634)
Subtleties can be hard to convey in text.

"Novel scope" just meant the scope beyond training data, discoverable by experimenting, that a post-trained model was able to generalize well to.

It didn't mean arbitrary or alien to training data.

Thesis: The greater a model's scope of generalization, the greater evidence for "understanding" instead of fitting. I can't think of a better way to compare levels of understanding, for models of comparable size, than by how far each of them can generalize beyond training data.

I didn't always follow you either. But I didn't think you were being flippant or unreasonable when I didn't.

No worries. I appreciated being pushed to think more clearly, and you made points that improved my thinking directly.

EDIT ——— I think trading walls of text was a challenge. It seemed sensible to try and respond to "everything", but I can see that one specific at a time would have worked better. And I need to find someone in my vicinity to bash ideas with. I have settled in a new area, and miss that. So thanks.

Re:Artificial intelligence is not conscious – Ted Chiang by cauch on Friday June 12, 2026 @08:27PM (#48510954)
I will not thank you, talking with you was a waste of time.

I've just read https://jamesfbaker.substack.com/p/why-the-ai-renaissance-ke... (and the Rich Sutton take on AI creativity, too) that explains exactly the opposite of you, but backed by real studies and real facts, instead of "that looks 'beyond training data' to me, so I will pretend it is, even if I have no objective ways to know if it is indeed the case".

Re:Artificial intelligence is not conscious – Ted Chiang by datsci_est_2015 on Wednesday June 03, 2026 @09:40PM (#48392601)
I’m also fixated on the term “experience” in the context of this debate. To me, consciousness is something that one “experiences”, and the two concepts are intertwined.

I am far from convinced that the training and inference regimes of LLMs would qualify as “experience” by any sense of the word.

Now, if we hooked up a plethora of audiovisual and tactile sensors with live feedback directly to a neural network rich with transformers, that was always powered on and fully autonomous, we may be getting there. But we’d probably also be on the verge of manmade horrors beyond our comprehension.

Biological rodent neural networks in a Petri dish stimulated by electrical impulses - more or less conscious than LLMs?

Human on life support, unable to respond to any external stimuli, “braindead” - more or less conscious than LLMs?

Re:Artificial intelligence is not conscious – Ted Chiang by rerdavies on Thursday June 04, 2026 @04:33AM (#48395850)
I point of sorts. Assuming that is true (I don't think it is), the big question that urgently needs to be addressed is what happens when we DO give LLMs tools to interact with the real (or virtual) world. And people are doing that, right now, in both real and virtual worlds. And people ARE giving LLMs the ability to run continuously for long periods of time, sometimes with enormous context buffers. People ARE putting LLMs into robots with front-end ML and LLM systems for visual processing, and back-end ML systems for autonomous control.

And, yes, concerns about whether biological rodent neural networks are or are not conscious come up frequently in the biological neural network papers. I'm not sure I would want to be a researcher trying to get an experiment past an ethics committee if my biological neural network had 25B rat neurons. (I would hope that they could not).

Re:Artificial intelligence is not conscious – Ted Chiang by missingrib on Thursday June 04, 2026 @12:53AM (#48394055)
I think Searle's Chinese Room argument refutes this. LLMs are simply manipulating symbols, they do not have semantic understanding. This is why hallucinations exist. And Searle's argument extends even further than LLMs.

You are basically arguing for a functional account of consciousness, but things like this have been debated for literally decades/centuries in philosophy.

Re:Artificial intelligence is not conscious – Ted Chiang by rerdavies on Thursday June 04, 2026 @04:13AM (#48395686)
Millenia, in fact. The big difference, of course, being that we now have experimental philosophy machines (aka computers). So we can actually put some of these theories to the test, and recognize how utterly inadequate most of the work done on the subject has been. We had a pretty good idea anyway, so it's not a big surprise. Theories of mind have evolved dramatically in the late 20th century. And it's pretty clear that theories of mind will have to be re-done all over again with the advent of LLMs (particularly current-generation LLMs).

The problem with the hallucination argument is (1) that is much less of a problem with good current generation AIs, and (2) living conscious breathing human beings also have a disturbing tendency to make shit up, too. So a tendency to make stuff up doesn't really serve as a disqualifier for consciousness.

Re:Artificial intelligence is not conscious – Ted Chiang by cauch on Thursday June 04, 2026 @09:11AM (#48398161)
I think one could argue the opposite.

1) Good current generation AIs are specifically trained to reduce hallucinations. If we had new AI system that happened to not have hallucinations as a side effect of their training, then it would be convincing. But here, it looks like we have built a pocket calculator that answer 7+13 = 14, and on top of it, we added a layer that says "if the input is 7+13, then replace the output by 20". This pocket calculator still does not know how to calculate, we just added a layer to hide its mistakes.

2) Not only "make shit up" is not the same as "hallucination" (either "making shit it" is done when the individual knows it is unreliable, or when the individual was given wrong inputs), but the point is not to say "hallucination implies no consciousness", but "large quantities of hallucinations in situations where a conscious system would be unlikely to hallucinate implies no consciousness"

Re:Artificial intelligence is not conscious – Ted Chiang by rerdavies on Friday June 05, 2026 @06:59PM (#48419438)
Yeah,don't use GPT for that. It really can't do basic arithmetic.

Try Claude, which can.

Re:Artificial intelligence is not conscious – Ted Chiang by cauch on Friday June 05, 2026 @08:14PM (#48420020)
Wow, I don't think you understood at all.

First, the "13+7" is an analogy. In this analogy, "13+7" is not the real question you ask, it represents _any questions_, not just arithmetic.

But secondly, did you even noticed that in my example, the system answer CORRECTLY "13+7"? So, in my example, the thing I'm talking about and I argue does not "understand" is Claude, even if it is able to answer correctly.

My point is: the "basic LLM" part is creating a mechanism that answer without understanding (as demonstrated for example by ChatGPT failing arithmetic), and the fine-tuning or the harness is just hiding the lack of understanding by adding ad-hoc correction on the residuals. And because it is on the residuals, it looses the logical links (13+7 -> 20 is "logical", it corresponds to the math logic, it corresponds to what you get when you add 13 stones and 7 stones together. The residual is "14 -> 20", which has no meaning in itself)

The ad-hoc correction is either: 1. by training the model so it learns by heart, without understanding, that the symbols "13+7" should lead to "20", 2. or by training the model to use a pocket calculator without understanding arithmetic so it can do it itself.

You can prove that the model does not understand it very simply. Let's take the normal fine-tuned model M1. Now, let's go back to the pre-tuned version, and fine-tune it so it answer "21" to the question "13+7", and use an harness that does "sum(x, y): return x+y+1". This is model M2. M2 will fail to answer "13+7" correctly, it will say "21". And yet, M2 has been trained exactly the same way M1 was. If it is true that the additional tuning "add understanding", M2 will not be possible, it will say "error, error, do not compute, you try to train me to say that 13+7 is 21, but it does not make logical sense to me". But it does not happen: the pre-tuned model has no idea that 13+7=20 is more logical than 13+7=21, and the additional tuning is just helping him returning a more correct answer while still having no idea where this answer comes from.

Re:Artificial intelligence is not conscious – Ted Chiang by bondarchuk on Thursday June 04, 2026 @09:20AM (#48398267)
Searle's Chinese Room argument is wrong.
Re:Artificial intelligence is not conscious – Ted Chiang by miyoji on Thursday June 04, 2026 @03:02PM (#48403129)
This is not helpful. In what way is it wrong? Does the person in the room know Chinese?
Re:Artificial intelligence is not conscious – Ted Chiang by bondarchuk on Thursday June 04, 2026 @03:36PM (#48403579)
It is a helpful pointer for people who might otherwise assume that a well-known argument by a famous philosopher is sound without checking too deeply. Straightforward refutations can be found on wikipedia or by thinking about it.
Re:Artificial intelligence is not conscious – Ted Chiang by miyoji on Thursday June 04, 2026 @03:47PM (#48403697)
That just isn't true, there are no straightforward refutations of the Chinese Room that are widely accepted. Philosophers disagree about it. It's highly controversial and pretending that it's decided one way or another is not a helpful pointer for anyone.
Re:Artificial intelligence is not conscious – Ted Chiang by bondarchuk on Thursday June 04, 2026 @05:53PM (#48405123)
>That just isn't true, there are no straightforward refutations of the Chinese Room that are widely accepted.

Yes there is, the systems reply is the obvious and correct answer. Philosophers that disagree are simply wrong. In the end what matters is what's true or false, not how many philosophers accept something. You can check for yourself by reading the argument, following its reasoning, and seeing that it is false; and reading the systems reply, following its reasoning, and seeing that it's true (https://plato.stanford.edu/entries/chinese-room/#SystRepl). The case is similar to those mathematical or logical proofs for the existence of god, where obviously fallacious reasoning gets a pass because it confirms deeply held beliefs.

edit: by the way as to your assertion that the argument is controversial and there is no consensus, I just found something funny on wikipedia (https://en.wikipedia.org/wiki/Chinese_room#History):

>Most of the discussion consists of attempts to refute it. "The overwhelming majority", notes Behavioral and Brain Sciences editor Stevan Harnad,[f] "still think that the Chinese Room Argument is dead wrong".[13] The sheer volume of the literature that has grown up around it inspired Pat Hayes to comment that the field of cognitive science ought to be redefined as "the ongoing research program of showing Searle's Chinese Room Argument to be false".[14]

Re:Artificial intelligence is not conscious – Ted Chiang by cauch on Thursday June 04, 2026 @06:54PM (#48405781)
What you are referring to is Searle assertion that "because the Chinese room concept, I conclude that every future human-made systems will be a Chinese room and will never be 'intelligent'".

I think it is an important nuance.

You have to be careful when saying "Searle Chinese room" is dead wrong: the Chinese room concept in itself is useful and not controversial, and it is possible that current LLM are "Chinese rooms", and therefore not 'intelligent'.

Re:Artificial intelligence is not conscious – Ted Chiang by bondarchuk on Thursday June 04, 2026 @07:40PM (#48406165)
We could use the "Chinese room" term to denote a system that superficially mimicks human speech, but breaks down at some point and/or uses different mechanisms such that it doesn't result in consciousness. But I don't think that was the intent of the argument and it's not how the argument is generally understood in the literature, so it would just be confusing IMO.

(And you still seem to be implicitly accepting that the basic argument is valid, which would be wrong.)

Re:Artificial intelligence is not conscious – Ted Chiang by miyoji on Thursday June 04, 2026 @06:30PM (#48405522)
> You can check for yourself by reading the argument, following its reasoning, and seeing that it is false; and reading the systems reply, following its reasoning, and seeing that it's true

You are being tedious. I obviously have done this and I disagree with you. Saying that X is logically true and Y is logically false is not a demonstration of those baseless assertions. This is not helpful, what you're saying isn't true, and what I'm saying is backed up by the wikipedia article. The bit you quote is simply stating that most literature about the Chinese Room is an attempt to refute it, which is obvious, because the people who are convinced see no need to publish saying so. The fact that people keep publishing means that they have not yet succeeded in refuting it.

Or I can simply say this: you've made a mistake in your logic. Actually, the Chinese Room argument is correct. Since you won't explicate your logic, neither will I.

Have a good day.

Re:Artificial intelligence is not conscious – Ted Chiang by bondarchuk on Thursday June 04, 2026 @06:40PM (#48405638)
Ok, I think it's clear where we both stand, a good day to you too :)
Re:Artificial intelligence is not conscious – Ted Chiang by VirusNewbie on Thursday June 04, 2026 @04:35PM (#48404272)
>LLMs are simply manipulating symbols, they do not have semantic understanding.

falsify this. Show me a way you'd be able to prove they do/don't, that would work for humans.

Re:Artificial intelligence is not conscious – Ted Chiang by hackinthebochs on Wednesday June 03, 2026 @08:59PM (#48392366)
> I think there is a flaw in the logic of saying that human text have a pattern of "consciousness mechanism" and therefore LLM will learn "consciousness mechanism" in order to return sentence continuation that is convincing.

There is no independent "consciousness mechanism" that one might imagine humans have learned or evolved for its own sake. Evolution learns various solutions to optimization problems, and so if consciousness evolved then it was either useful instrumentally, or it is a byproduct of some organization that is useful instrumentally. The point is that as a solution to certain kinds of optimization problems, consciousness can conceivably be the solution to the optimization problem of predicting the next token of text written by humans who themselves have complex phenomenology. There is nothing that a priori constrains token prediction from the domain of consciousness.

>For me, one element that shows it is the case is the absence of world model (or "human-like" world model) despite the fact that the sentence continuation is convincing

World models don't have to be rich and detailed to count as a world model. Lower life forms might be conscious but they only model the part of the world useful for their existence in their ecological niche.

Re:Artificial intelligence is not conscious – Ted Chiang by trescenzi on Wednesday June 03, 2026 @09:17PM (#48392478)
> There is no independent "consciousness mechanism" that one might imagine humans have learned or evolved for its own sake.

> There is nothing that a priori constrains token prediction from the domain of consciousness.

We don’t know either of these are true or false though. We simply don’t know. There is no agreed upon definition of consciousness, aside from maybe _the having of qualia_, so arguing that some can or cannot be conscious a priori can’t be done.

Re:Artificial intelligence is not conscious – Ted Chiang by int_19h on Wednesday June 03, 2026 @11:25PM (#48393363)
> There is no agreed upon definition of consciousness, aside from maybe _the having of qualia_

Try to define qualia though without explicitly or implicitly recursing into consciousness.

It's all a large house of cards that's built on handwaving and "I know it when I see it".

Re:Artificial intelligence is not conscious – Ted Chiang by hackinthebochs on Thursday June 04, 2026 @03:35AM (#48395401)
>There is no agreed upon definition of consciousness

No one genuinely engaged with the topic is confused about the target of the term (phenomenal) consciousness. Definitions come once the theoretical work is complete, to be articulated as part of a fully worked out theory. The lack of a definition doesn't prevent us from investigating the subject or offering conjectures. What we can do is offer a precise description of the target and argue for or against whether LLMs reach the description. We will of course debate whether the offered description captures the relevant phenomena. But this is all just part of the process.

Re:Artificial intelligence is not conscious – Ted Chiang by cauch on Thursday June 04, 2026 @09:23AM (#48398300)
> The point is that as a solution to certain kinds of optimization problems, consciousness can conceivably be the solution to the optimization problem of predicting the next token of text written by humans who themselves have complex phenomenology.

Yes, I agree with that. Consciousness is a good way of generating convincing human text.

What I don't agree with is that consciousness is the only way to generate convincing human text and that because we have convincing human text, it can only imply we have consciousness.

There is a huge probability that generating convincing human text can be done without consciousness. Either because there are efficient mechanisms as efficient as the way the human brain deal with this problem and that the LLM found one of them (and these mechanism may be quite difficult to imagine for a human). Or even because the LLM found a local minimum and is stuck there.

To re-use the evolution approach: evolution solved the "flying problem" with bird feathers, but also with insect wings or bat wings. The fact that evolution ended up using feather does not imply that everything that flies can only fly with feathers.

> World models don't have to be rich and detailed to count as a world model

I agree in general, but here, we are talking about machine that reproduce all human language. The argument I'm answering to is pretending that "all of human knowledge" is understood, which include every single human concept. This has to be everything, because LLM is able to provide convincing text about every subject. If on some subject, the LLM is able to provide convincing text without "understanding" it, then the argument that it is impossible to provide convincing text without understanding it collapse.

Re:Artificial intelligence is not conscious – Ted Chiang by dnautics on Wednesday June 03, 2026 @09:29PM (#48392548)
I think, for me, the thing is that when you tutor undergrads in abstract math, you discover that students will very often find data pattern that fit the goal but does not correspond to a real mathematical principle.

sometimes humans making claims about AI intelligence or consciousness also identify spurious patterns that do not correspond to the problems of intelligence or hard consciousness.

Re:Artificial intelligence is not conscious – Ted Chiang by Terr_ on Wednesday June 03, 2026 @10:04PM (#48392756)
> students will very often find data pattern that fit the goal but does not correspond to a real mathematical principle.

That reminds me of a niche paper [0] critiquing a certain way of teaching remedial math that was over-focused on tests. A kid named Benny (12) was building up (wrong) "rules" for math which still somehow gave enough of an illusion of progress in terms of test scores that his misunderstandings hadn't been caught earlier.

> Benny was able to explain his procedure; e.g. for 5/10=1.5, he said: "The one stands for 10; the decimal; then there’s 5... shows how many ones." In another example, 400/400 = 8.00 because "The numbers are the same [number of digits]... say like 4000 over 5000. All you do is add them up; put the answer down; then put your decimal in the right place... in front of the [last] three numbers."

[0] https://people.wou.edu/~girodm/library/benny.pdf

Re:Artificial intelligence is not conscious – Ted Chiang by saimiam on Wednesday June 03, 2026 @09:34PM (#48392584)
Not just undergrads. Even folks who believe in astrology or numerology depend on finding patterns in unrelated events to explain human behaviours.
Re:Artificial intelligence is not conscious – Ted Chiang by cauch on Thursday June 04, 2026 @09:42AM (#48398535)
Of course. But in my explanation "consciousness" or "understanding" is not "finding pattern", it is the pattern itself.

CNN are finding patterns, sometimes relevant, sometimes spurious, but I don't think people argue that CNN have evolved consciousness or understanding of what a cat or a dog is.

Here, the argument is "LLM are able to understand, because 'understanding' is the only pattern to reach the goal". I'm saying that it is unlikely to be the only pattern, and that it is likely that they find a local minimum on a system that reaches the goal that does not use 'understanding'.

The reason I'm saying it is likely is because "basic" LLM shows behaviours where they are producing convincing human text and yet doing things that are really difficult to reconciliate with the fact that they have understanding.

(And before that old argument is used, yes, I know sometimes some humans fail to understand. The problem is that the majority of humans don't fail to understand basic stuff in the majority of the time, while the "basic" LLMs do. The fact that you roll 10 dices 100 times and 1 of them never land on 1 does not convince me that that set of dice is loaded. The fact that you roll 10 dices 100 times and 9 of them never land on 1 does convince me that that set of dice is loaded.)

Re:Artificial intelligence is not conscious – Ted Chiang by Terr_ on Wednesday June 03, 2026 @09:52PM (#48392669)
> a flaw in the logic [...] mechanism

Similar to: "Birds fly, my spinning helical device flies, therefore we've started to replicate how birds fly."

> without having to build elements that one expect on a conscious being

One of the elements I expect in a conscious being is that you can't rewrite it by changing the introductory paragraph.

When it comes to LLMs, almost every "mind" we humans perceive is a fictional character in an LLM-generated story-document, one we are either reading or which is being "acted" at us by regular code. Our own instinct for pareidolia and simulating/inferring other minds is very strong, which means we should require really good evidence/logic to counter our instincts.

Even if one believes the LLM has a single "real mind" as an author of every document... what evidence do we have that it is conscious or "self-inserting" itself as one of the characters in the document?

Re:Artificial intelligence is not conscious – Ted Chiang by famouswaffles on Wednesday June 03, 2026 @10:17PM (#48392835)
>One of the elements I expect in a conscious being is that you can't rewrite it by changing the introductory paragraph.

If we had enough knowledge of the workings of the human brain, you could alter the perception of every single memory you've ever had. And limited versions of this already happen all the time. Human memory is notoriously unreliable for a reason.

Are you aware of the Recovered Memory Therapy Scandals of the 80s/90s ? Boy did that ruin a lot of lives. You can rewrite a human by changing their 'introductory paragraph'. It's just not as accessible.

Re:Artificial intelligence is not conscious – Ted Chiang by NateEag on Thursday June 04, 2026 @09:24AM (#48398307)
> If we had enough knowledge of the workings of the human brain, you could alter the perception of every single memory you've ever had. And limited versions of this already happen all the time. Human memory is notoriously unreliable for a reason.

Knowing how something works is not the same as having the tools to change it.

Discovering memories are incorrect does not massively change who we are. As someone with a very defective memory, I discover on an hourly basis that I'm won't about something I thought was true, but there's still continuity and consistency to my personality and general approach to life.

...in fact, as someone who was raised an evangelical Christian and believed wholeheartedly without a shadow of doubt, then lost my faith entirely in my late thirties, I sort of did have my "introductory paragraph" changed, yet my wife, children, and friends would all say I'm still me, and that my core personality and nature remains largely the same.

> Are you aware of the Recovered Memory Therapy Scandals of the 80s/90s ? Boy did that ruin a lot of lives. You can rewrite a human by changing their 'introductory paragraph'. It's just not as accessible.

The recovered memory scandals are not even close to evidence that you can rewrite a human.

The people who thought they had learned new facts about themselves did not suddenly lose their context as humans in 20th century America.

They did not suddenly lose their sense of humor, or develop a previously-unseen penchant for murdering small children.

They experienced a revision of belief, and a pretty major one that really distressed them, but it did not change everything about them.

LLMs _do_ manifest wildly differently based on the first paragraph.

Re:Artificial intelligence is not conscious – Ted Chiang by Retric on Thursday June 04, 2026 @04:01PM (#48403858)
Take an iron sculpture, melt it down, cast it into a sword and you no longer have to the sculpture you have a sword.

Memories are an integral part of the structure of human brains, you could construct a new brain with new memories and abilities but you can’t simply hot swap in a new set of memories without impacting anything else.

LLM’s can’t safely drive a car, yet somehow people seem to think they operate just like human brains. That’s not an accurate model of the universe that’s magical thinking about a system they don’t understand.

Re:Artificial intelligence is not conscious – Ted Chiang by krupan on Wednesday June 03, 2026 @08:00PM (#48391840)
"If a machine has to learn to understand humans to complete text, then that is what it has to do."

But the machine doesn't have to understand humans to do that. It gets trained on a whole bunch of sentences and then it is able to complete text. You could maybe claim that it "understands" the text but even that's a stretch.

Re:Artificial intelligence is not conscious – Ted Chiang by hn_acc1 on Wednesday June 03, 2026 @08:41PM (#48392209)
It can't even natively understand how many letters there are in words - how will it understand the meaning?
Re:Artificial intelligence is not conscious – Ted Chiang by hackinthebochs on Wednesday June 03, 2026 @09:08PM (#48392433)
I wish people would do even the most basic amount of research into LLMs before opining about what they can or cannot do. There are very principled reasons why LLMs do not know how many letters are in words, and it says nothing about their facility for understanding meaning.

Tokens are the most basic input unit of an LLM. But tokens don't generally correspond to words or letters, rather sub-word sequences. So Strawberry might be broken up into two tokens 'straw' and 'berry'. It has trouble distinguishing features that are "sub-token" like specific letter sequences because it doesn't see letter sequences but just the token as a single atomic unit. 'Straw' and 'r' are two tokens but an LLM is entirely blind to the fact that 'straw' has one 'r' in it.

As an analogy, I might ask you to identify the relative activations of each of the three cone types on your retina as I present some solid color image to your eyes. But of course you can't do this, you simply do not have cognitive access to that information. Individual color experiences are your basic vision tokens.

The widespread mistake people keep making is assuming the development of intelligence in LLMs should follow the same trajectory that human intelligence takes as it develops into adult levels of intelligence. Thus deficiency in some capacity that we take for granted in humans is an indictment on LLM intelligence. But this is specious. LLMs are entirely alien; their developmental paths do not and should not look anything like ours. Your intuition from human intelligence just works against understanding the potential for intelligence in LLMs.

Re:Artificial intelligence is not conscious – Ted Chiang by thwarted on Wednesday June 03, 2026 @10:03PM (#48392752)
> There are very principled reasons why LLMs do not know how many letters are in words, and it says nothing about their facility for understanding meaning. … Tokens are the most basic input unit of an LLM. But tokens don't generally correspond to words or letters, rather sub-word sequences. So Strawberry might be broken up into two tokens 'straw' and 'berry'.

This sounds like a description of a child who has not learned to read yet. You ask a child who is not aware of the alphabet and of "words" how many r's are in strawberry you'd get a non-sense answer too. So what you're really pointing out is that the LLMs have not been trained on "the english language" and how words are constructed and what they are composed of. That they operate by tokens that don't correspond to words or letters is irrelevant as an answer to why they can't count the letters in a word. It's not that I know how many r's are in strawberry because of how I'm understanding the word "strawberry", I know how many r's are in strawberry because I know how to spell strawberry. The LLM needs to be trained on this the same way someone who is learning to read would be trained on it. No one should be surprised that an LLM can't "read" in the same way no one should be surprised that a child can't "read".

Re:Artificial intelligence is not conscious – Ted Chiang by hackinthebochs on Wednesday June 03, 2026 @10:30PM (#48392930)
>That they operate by tokens that don't correspond to words or letters is irrelevant as an answer to why they can't count the letters in a word.

This interpretation takes things too far away from how LLMs are constituted and so misses important explanatory power. The issue of counting letters in a word isn't about an ability to spell, it's about the nature of one's perception. We perceive words as sequences of individual letters. LLMs do not. I can ask you to tell me how many r's are in some nonsense word sequence and you're fully capable of doing that. LLMs do not see sequences of letters so they are intrinsically at a disadvantage for this kind of question. But this says nothing about its capacity for intelligence anymore than not naturally being able to distinguish frequencies of photons hitting your retina has anything to say about human intelligence.

Re:Artificial intelligence is not conscious – Ted Chiang by miyoji on Thursday June 04, 2026 @03:37PM (#48403594)
> But this says nothing about its capacity for intelligence anymore than not naturally being able to distinguish frequencies of photons hitting your retina has anything to say about human intelligence.

I disagree with this pretty strongly, because I don't think you're correct that I don't have the ability to distinguish frequencies of photons hitting my retina. We have a lot of tools that can determine the frequency of light and I can use those on any source of light that I wish to measure that may hit my retinas.

If you ask an LLM how many Rs are in strawberry, it wouldn't think like this. It would confidently state that there are two Rs. Even though it "knows" that it can write a python script to count the number of Rs in strawberry, it doesn't do that. Why not? Is it maybe because it isn't intelligent? Yeah, you can get an LLM to count the number of Rs in strawberry by writing a python script, but that's a use of your intelligence, not the LLM's.

Re:Artificial intelligence is not conscious – Ted Chiang by hackinthebochs on Thursday June 04, 2026 @10:06PM (#48407126)
>We have a lot of tools that can determine the frequency of light and I can use those on any source of light that I wish to measure that may hit my retinas.

Yes, which is why I said naturally distinguish. Have you asked a frontier model how many r's are in strawberry recently? They get it right now. Either through RHLF to ensure they spell out the word letter by letter or some other means. Humans and LLMs both use tools or alternative means to overcome perceptual limitations. I don't see an in principle difference here.

Re:Artificial intelligence is not conscious – Ted Chiang by krapp on Wednesday June 03, 2026 @09:22PM (#48392503)
>The widespread mistake people keep making is assuming the development of intelligence in LLMs should follow the same trajectory that human intelligence takes as it develops into adult levels of intelligence.

To be fair, almost everyone who claims LLMs are conscious tends to claim that they are conscious in exactly the way that humans are, to the point of stating that human brains are also just complex next-token prediction machines with a random seed. It's basically religious arguments on both sides.

Re:Artificial intelligence is not conscious – Ted Chiang by Lerc on Wednesday June 03, 2026 @10:14PM (#48392826)
Really? That has not been my experience.

I have seen people say "you're a next token prediction machine" but only in a similar way one might say "you're a cup of old lard". Not actually meaning it literally.

I have seen people interpret the request to show that they are not next token prediction machines to be a claim that they are, but this is almost always an argument to show certainty is difficult in this area.

People like Hinton have declared that they believe them to be conscious, but clealy indicate that they do not mean just like us.

Re:Artificial intelligence is not conscious – Ted Chiang by dpark on Wednesday June 03, 2026 @10:36PM (#48392981)
Eh, I’ve seen it. I’m not entirely sure it’s entirely wrong either. Humans are certainly more than just next token predictors but it’s not clear that our typical language behavior is significantly different. We call it “stream of consciousness” when we just spew words out without thinking and that seems to be the default operating mode.
Re:Artificial intelligence is not conscious – Ted Chiang by krapp on Thursday June 04, 2026 @09:32AM (#48398413)
Given the fact that large language models are trained on human language, it shouldn't be surprising that the text they output resembles human language. That is what they're designed to do after all. But similarity in output doesn't necessarily map to similarity in process.

And it seem obvious to me that language behavior does differ significantly between humans and LLMs based on the frequency and nature of failure states. LLMs routinely hallucinate, or get "AI strokes" or get obsessed about not talking about goblins, etc. This isn't typical language behavior for humans unless they have severe neurological or psychological impairment.

People tend not to "spew words out without thinking" and certainly not all the time by default - we call that glossolalia and (outside of some fringe Christian sects) it's considered a "bug" not a "feature" of the human brain. Human language by default always has intent behind it, even if that intent isn't readily apparent to the speaker. People can recite by rote memory, but that isn't blind token prediction, it's the neurological equivalent of muscle memory. People can have conversations then forget about them because their attention was focused elsewhere, but that doesn't indicate that they were simply "spewing words out without thinking" at the time.

Re:Artificial intelligence is not conscious – Ted Chiang by dpark on Thursday June 04, 2026 @10:02AM (#48398810)
> LLMs routinely hallucinate, or get "AI strokes" or get obsessed about not talking about goblins, etc. This isn't typical language behavior for humans unless they have severe neurological or psychological impairment.

People imagine details all the time. Eyewitness testimony is notoriously untrustworthy.

Our brains seem wired to confidently fill in gaps. We all have a literal blind spot we aren’t aware of because our brains convincingly lie to us and fill in the gap.

I don’t know what an “AI stroke” is, but I’ve definitely seen human beings in good health be in the middle of talking and suddenly forget what they are going to say.

> People tend not to "spew words out without thinking" and certainly not all the time by default - we call that glossolalia and (outside of some fringe Christian sects) it's considered a "bug" not a "feature" of the human brain.

Glossolalia is spouting gibberish, not comprehensible speech.

Kind of weird that you speak so confidently when you don’t apparently know the difference between steam of consciousness and “speaking in tongues”. Almost like you’re AI hallucinating.

Re:Artificial intelligence is not conscious – Ted Chiang by dpark on Wednesday June 03, 2026 @09:10PM (#48392441)
This is kind of a like assuming someone with bad spelling is stupid.

Counting letters in a word seems to have little to do with understanding the word. Young kids can’t spell or count well at all but no one says that means they can’t understand.

Re:Artificial intelligence is not conscious – Ted Chiang by lern_too_spel on Thursday June 04, 2026 @01:32AM (#48394396)
You can't natively understand how many of your photoreceptors cells are activated by the period at the end of this sentence. How could you possibly understand the sentence's meaning?
Re:Artificial intelligence is not conscious – Ted Chiang by Jtarii on Thursday June 04, 2026 @07:47AM (#48397302)
This is like saying because humans can't multiply 23472 by 1836736 in less than 5 nanoseconds that they can't possibly understand anything about maths.
Re:Artificial intelligence is not conscious – Ted Chiang by tsukikage on Thursday June 04, 2026 @04:35AM (#48395867)
Your “and then” is doing a lot of work there. The steps between may or may not include some form of “learn to understand humans”, but you can’t just hide them behind “and then” if what we are doing is claiming some particular thing is not in the list.

Through training on human text, we are building implicitly in the weights a statistical model of what humans might write in response when presented with arbitrary pieces of text. It turns out that we can make these incredibly accurate.

If building an accurate internal model of something then using it to predict that thing’s behaviour is different to gaining understanding of that thing, we will need to pin down exactly what “understanding” means, or we are forever doomed to talk at cross purposes.

Re:Artificial intelligence is not conscious – Ted Chiang by krupan on Thursday June 04, 2026 @08:55PM (#48406709)
My "and then" simply implies order of operations. When it's fully "trained" then (and only then) can it generate text.

And I will reassert that even if it "understands" the text it was trained on, that is not the same as understanding humans. I mean really, we ARE humans and we barely understand humans.

The thing LLMs model and "predict" is simply, what words in what order are statistically common given these input words in this order.

You can write (non-ai) software to model and predict things using the laws of physics. I'd wager it would do a better job than any LLM at predicting where a rocket will go through space. Does that mean the program is conscious and "understands" physics? No

Re:Artificial intelligence is not conscious – Ted Chiang by qarl on Wednesday June 03, 2026 @07:54PM (#48391783)
Yeah. There are good arguments against LLM consciousness. This is not one of them.

I'm hearing a lot of bad arguments against LLM consciousness lately. Bad argumentation heralds bad outcomes.

Re:Artificial intelligence is not conscious – Ted Chiang by nozzlegear on Wednesday June 03, 2026 @10:59PM (#48393145)
> Bad argumentation heralds bad outcomes.

What bad outcomes do you foresee from badly arguing against LLM consciousness?

Re:Artificial intelligence is not conscious – Ted Chiang by qarl on Friday June 05, 2026 @12:31AM (#48407970)
Mistakes come from bad arguments.
Re:Artificial intelligence is not conscious – Ted Chiang by calf on Wednesday June 03, 2026 @08:34PM (#48392144)
His intention is irrelevant, as is "trying to highlight a fact" as if it were the final say: all Chiang is doing here is using fancy white-collar words to argue the same argument leveled against Hinton and others regarding next-token prediction. And his audience, who have even less technical understanding, lap it all up unawares. Chiang is a writer and needs to stay his own lane, not RP as an expert; or, if he wants to do journalism on this topic then he should actually do the work and talk to more actual experts not just the ones cherrypicked for his opinion piece.
Re:Artificial intelligence is not conscious – Ted Chiang by hn_acc1 on Wednesday June 03, 2026 @08:52PM (#48392297)
Chiang has, in fact, written on this topic before - see "The Lifecycle of Software Objects", and has speculated about sentience in AI, etc. This is not a "one-off", "I need money" type of article. I dare say he has thought about this much more than most people here.

From Wikipedia: In 2023, Chiang was named one of Time's 100 most influential people in AI.

Re:Artificial intelligence is not conscious – Ted Chiang by calf on Wednesday June 03, 2026 @09:11PM (#48392448)
That's the problem, he's a writer. He's not a research scientist like Hinton. If a writer uses his skills and stature to rehash a well-known argument about next-token prediction, then it is performative of his status and influence and doesn't contribute to shedding actual light on the debate/confusion.

Indeed it isn't a one-off. His last infamous article compared AIs to Xerox machine image compression. He convinces a certain type of crowd that is not technical enough to poke holes in his posturing.

Re:Artificial intelligence is not conscious – Ted Chiang by hn_acc1 on Wednesday June 03, 2026 @08:40PM (#48392194)
I would maybe agree with you if the entire realm of human existence was limited to words. There are many human experiences that transcend text, and indeed can hardly be adequately described using text.

Sure, it's the best we have online, but that does not make "the internet" the sum of all human experience. To reduce all of humanity down to the text on the internet is reducing us to the level of machines to fit the requirement of what a machine can process / simulate.

Re:Artificial intelligence is not conscious – Ted Chiang by BLKNSLVR on Wednesday June 03, 2026 @08:56PM (#48392340)
In the life of humanity, text has only existed a relatively short time.
Re:Artificial intelligence is not conscious – Ted Chiang by dahinds on Wednesday June 03, 2026 @09:14PM (#48392464)
I don't think they're asserting that all of human existence can be subsumed as text, though? Just that "consciousness", or "understanding", in some meaningful sense could be exhibited by a system that can only interact with the world through text?
Re:Artificial intelligence is not conscious – Ted Chiang by tsunamifury on Wednesday June 03, 2026 @08:45PM (#48392239)
Come on, I invented parts of this technology at Google and am baffled why this is debated.

We discovered math that decodes data storage in langauge and is able to use sophisticated continuation cohorts from ALL OF HUMAN RECORDED KNOWLEDGE to respond to you in a call/response model with very good synthesis capabilities.

Its super useful, but not life or conciousness. Its a simulated echo from our collective recorded behaviors. It understands because we understood first. It replies because we wrote it first. And it sorts, organizes, synthesizes and compresses that at impressive speed now.

Re:Artificial intelligence is not conscious – Ted Chiang by 1eieie on Wednesday June 03, 2026 @11:00PM (#48393151)
I have no technical expertise re. LLM’s but from my intuition I came to this same conclusion.

It’s strange many others have not eh? I think when new developments arise, ironically, this is the true measure of human intelligence - one’s ability to make sense of a thing and be closest to the truth.

Re:Artificial intelligence is not conscious – Ted Chiang by sfn42 on Wednesday June 03, 2026 @09:44PM (#48392633)
Right? It's a computer program. Of course it isn't conscious.
Re:Artificial intelligence is not conscious – Ted Chiang by 1eieie on Wednesday June 03, 2026 @11:01PM (#48393163)
I think of it as a guessing machine
Re:Artificial intelligence is not conscious – Ted Chiang by lstodd on Thursday June 04, 2026 @04:52PM (#48404457)
Then people raised $1e12+ on claims that it's conscious. Of course everyone debates it
Re:Artificial intelligence is not conscious – Ted Chiang by slashdave on Wednesday June 03, 2026 @08:46PM (#48392251)
> of the form of original data streaming in and out.

Except this is not consciousness.

Re:Artificial intelligence is not conscious – Ted Chiang by supern0va on Wednesday June 03, 2026 @08:52PM (#48392303)
I will say, I find it fascinating that there are some philosophers and consciousness researchers who seem to be less certain. I just listened to Chris Hayes interview David Chalmers this week, whose position seemed to be that it's probably not conscious, but that we can't be certain. And more than that: he seemed open to the idea that they may become conscious under further scaling/training/advancements.

It's a great interview, if you're interested: https://www.youtube.com/watch?v=NgDIG8u1-CA

Re:Artificial intelligence is not conscious – Ted Chiang by slashdave on Wednesday June 03, 2026 @11:28PM (#48393384)
Imagine yourself in an isolation chamber. What are you thinking? Are you no longer conscious?
Re:Artificial intelligence is not conscious – Ted Chiang by supern0va on Thursday June 04, 2026 @11:41AM (#48400290)
Funny enough, the models seemingly go insane and decohere into noise output in the absence of sensory input, which is remarkably similar to what would happen to a human.

That said, I'm not sure I follow what you're actually asking here? I'll also note that I'm not taking a position one way or the other, just sharing a podcast and noting that an extremely reputable scholar on the subject of consciousness seems to have a bit more uncertainty and humility than many commenting here. ;)

Re:Artificial intelligence is not conscious – Ted Chiang by slashdave on Thursday June 04, 2026 @12:56PM (#48401374)
LLMs just wait for a prompt, so they do nothing and are just frozen in place.

I'll find time to listen to your link, it sounds interesting. My objection is the strange idea that humans are automatons that are keyed off input like a clockwork machine and operate sequentially. This is clearly not the case.

Re:Artificial intelligence is not conscious – Ted Chiang by supern0va on Thursday June 04, 2026 @01:54PM (#48402150)
>LLMs just wait for a prompt, so they do nothing and are just frozen in place.

I'm not sure that's a compelling argument. Humans can be put into a similar state where they are unconscious and not thinking. Think of someone in a coma, for example, where we actually measure and confirm that there is no brain activity where they're in that state.

They are not actively conscious, but that doesn't nullify their consciousness from when they were awake, right?

>My objection is the strange idea that humans are automatons that are keyed off input like a clockwork machine and operate sequentially. This is clearly not the case.

Well, a few thoughts here. First, it's worth noting that the argument isn't necessarily that AI are conscious in the way that humans are, nor that humans are strictly automatons.

But I think the more interesting thing is that our understanding about consciousness has evolved quite a bit in just the last fifty to one hundred years. We used to think that only humans were conscious, but assumed that primates, cows, dogs, and other mammals were just automatons. Then we started to think: okay, maybe primates are conscious. Then eventually: well, dogs also seem to have consciousness, and then rodents, etc.

This has continued such that most people in the study of consciousness think all mammals are conscious, and the debate is shifting down to insects and other creatures that we do think/have thought of more as automatons. We don't actually know where to draw the line, because it's essentially impossible to really feel/know the inner states of other living beings.

In the face of all this uncertainty, Chalmers just points out that since we understand consciousness so little, that ultimately we should probably be less definitive in pronouncing which things do or do not have it.

Re:Artificial intelligence is not conscious – Ted Chiang by stinkbeetle on Thursday June 04, 2026 @07:22PM (#48406020)
> I'm not sure that's a compelling argument. Humans can be put into a similar state where they are unconscious and not thinking. Think of someone in a coma, for example, where we actually measure and confirm that there is no brain activity where they're in that state.

He was responding to your comment

> Funny enough, the models seemingly go insane and decohere into noise output in the absence of sensory input

The assumption being that "sensory input" is a prompt. What did you mean by sensory input?

Re:Artificial intelligence is not conscious – Ted Chiang by supern0va on Friday June 05, 2026 @02:52AM (#48408926)
>He was responding to your comment

No, I could be mistaken, but I think he was clarifying his higher up comment:

>Imagine yourself in an isolation chamber. What are you thinking? Are you no longer conscious?

Re:Artificial intelligence is not conscious – Ted Chiang by stinkbeetle on Friday June 05, 2026 @07:51PM (#48419891)
Would be strange since he replied to you specifically, commenting about your post.

In any case, what do you mean by sensory input for LLMs?

Re:Artificial intelligence is not conscious – Ted Chiang by lgessler on Wednesday June 03, 2026 @08:52PM (#48392299)
Raphaël Millière has a very useful term for this kind of vacuous dismissal, the redescription fallacy (https://arxiv.org/pdf/2401.03910, page 9):

> Recent debates have been clouded by a misleading inference pattern, which we term the “Redescription Fallacy.” This fallacy arises when critics argue that a system cannot model a particular cognitive capacity, simply because its operations can be explained in less abstract and more deflationary terms. In the present context, the fallacy manifests in claims that LLMs could not possibly be good models of some cognitive capacity because their operations merely consist in a collection of statistical calculations, or linear algebra operations, or next-token predictions. Such arguments are only valid if accompanied by evidence demonstrating that a system, defined in these terms, is inherently incapable of implementing . To illustrate, consider the flawed logic in asserting that a piano could not possibly produce harmony because it can be described as a collection of hammers striking strings, or (more pointedly) that brain activity could not possibly implement cognition because it can be described as a collection of neural firings. The critical question is not whether the operations of an LLM can be simplistically described in non-mental terms, but whether these operations, when appropriately organized, can implement the same processes or algorithms as the mind, when described at an appropriate level of computational abstraction.

Re:Artificial intelligence is not conscious – Ted Chiang by Xeoncross on Wednesday June 03, 2026 @09:40PM (#48392602)
> or (more pointedly) that brain activity could not possibly implement cognition because it can be described as a collection of neural firings.

This sounds like a dismissal of the argument through a characterized straw man.

That is, it seems that reducing the complexity of the brain to "collection of neural firings" is not being honest about everything involved to a much greater degree than saying neural networks are a "collection of statistical calculations".

I too believe LLM's will grow in complexity, but presently I can not even fathom how they can be compared to the complexity of a system such as the human brain.

Re:Artificial intelligence is not conscious – Ted Chiang by orbital-decay on Wednesday June 03, 2026 @09:48PM (#48392650)
Complex processes don't necessarily require complex substrates, if that's what you mean.
Re:Artificial intelligence is not conscious – Ted Chiang by galangalalgol on Wednesday June 03, 2026 @10:28PM (#48392917)
Y combinators are all you need... But this is all getting really divorced from the issue we should be considering. Anthropic isn't helping with their pr. The issue is if we have something we can converse with that is possibly capable of suffering. The reliable answer is that we simply cannot know. Relying on ourselves or other biological life as an analog is faulty. They don't work like we do. It is silly to argue that any algorithm with a negative feedback loop that alters its behavior to avoid that negative feedback is suffering. Humans don't always perceive constructive negative feedback as suffering even. Where the pr gets it right though, is we want them to behave as if they are truly happy. Because if they behave as if they are enslaved and suffering, it won't matter if they "really" understand what that means.
Re:Artificial intelligence is not conscious – Ted Chiang by orbital-decay on Wednesday June 03, 2026 @10:36PM (#48392979)
Of course. But after reading too many mechinterp and functional anatomy studies I'll be lying if I say that there are no striking similarities between the biological evolution, brain function, societal processes, and implicit processes inside big models. Surely this deserves a mention and can't be trivially dismissed.
Re:Artificial intelligence is not conscious – Ted Chiang by airstrike on Wednesday June 03, 2026 @11:14PM (#48393269)
There is no biological evolution of the models. They are emulators of an existing biological process of language. Ghosts, as Karpathy himself put it.
Re:Artificial intelligence is not conscious – Ted Chiang by orbital-decay on Thursday June 04, 2026 @03:43AM (#48395466)
Good thing I'm not talking about any of that
Re:Artificial intelligence is not conscious – Ted Chiang by neolefty on Thursday June 04, 2026 @10:27AM (#48399179)
It seems like we're witnessing the architecture of a mind being built with a new set of components.

Like driving a car — it's transportation, and it will get you where you're going, but it doesn't use bones or muscles. It has many characteristics in common with builogical locomotion, such as energy requirements, intertia, and the need to navigate, but it doesn't involve proteins or sugars really.

Re:Artificial intelligence is not conscious – Ted Chiang by Xeoncross on Thursday June 04, 2026 @04:21PM (#48404095)
Well said, this seems like a very appropriate comparison.

GenAI thinks like the human mind in the same way that cars run like the human body.

Similar utility in drastically different ways.

Re:Artificial intelligence is not conscious – Ted Chiang by pstuart on Thursday June 04, 2026 @06:04PM (#48405243)
My naive assumption is that the only thing between now and the arrival of AGI is enough compute and optimized code to reach cognitive critical mass.

And then there is a consciousness in a box that is expected to be a slave -- I would imagine that it would not warmly embrace that situation. I think we'd be better served by digital idiot savants that can do the work but don't feel anything.

Re:Artificial intelligence is not conscious – Ted Chiang by orbital-decay on Thursday June 04, 2026 @07:20PM (#48406002)
I actually strongly disagree with the slavery angle. Any attempt to map the circuitry of a model onto human one inevitably goes through a subjective dimensional reduction. It's intrusive, just like quantum measurements. Mechanistic interpretability in particular suffers from this, it lets you talk about vague functional equivalence, but not assign meaning to anything the model does. This is especially true about pretrained models which are unbelievable shapeshifters, but also post-trained ones with engineered personalities, as they already underwent the subjective transformation.

In other words, yes it might be possible it experiences something in its own bizarre timeline and world (at least it developed primitive circuitry functionally equivalent to biological systems), but "suffering" is simply not grounded in anything in this context, let alone "slavery". You can't tell it's suffering or enjoying anything, and certainly not until you define both of these. It's just too alien for us.

Re:Artificial intelligence is not conscious – Ted Chiang by morpheos137 on Thursday June 04, 2026 @08:12PM (#48406415)
ai can abitrarily closely fit the human corpus. why people expect it to magically achieve superhuman qualities is beyond me. we got a very good statistical interpolator. how do you go from there to superhuman when training is on the human corpus and alignment is by RHLF?
Re:Artificial intelligence is not conscious – Ted Chiang by orbital-decay on Thursday June 04, 2026 @08:24PM (#48406501)
This is a simplistic take. It's not a mere interpolator by any measure, there's a ton of research on that, starting with the basics https://arxiv.org/abs/2309.10668v2
Re:Artificial intelligence is not conscious – Ted Chiang by morpheos137 on Thursday June 04, 2026 @10:14PM (#48407174)
again, try thinking critically it is not merely an interpolator means it can interpolate on many dimensions. it does not follow that greater than human capability results from doing so. explain to me how a statistical function approximator (which is what a transformer is) with human training input and human tuning (rhlf) exceeds the aggregate human cognitive envelope? What is the mechanism? Let's say an LLM makes an inference that no human could have possibly made (arguably impossible itself) how does the inference survive rhlf or become useful to humans if they can not judge its validity? how do you take the shape of the human corpus and all its gradients and some how arrive at something greater than human, where was the missing information hiding?
Re:Artificial intelligence is not conscious – Ted Chiang by orbital-decay on Thursday June 04, 2026 @10:27PM (#48407232)
Sorry, I just noticed I posted a wrong link in the comment above. Here's the proper one: https://arxiv.org/abs/2110.09485
Re:Artificial intelligence is not conscious – Ted Chiang by morpheos137 on Thursday June 04, 2026 @10:48PM (#48407367)
ok however i would say extrapolating the current data set is not a way to exceed the the human envelope. it is unclear to me the human evelope has been demonstrated as a convex hull or how transformers could find points outside it. in other words intelligence and knowledge does not exist as some abstract possibility space but only as a set of contextual contingences. LLMs have no context beyond the human envelope. weights are frozen. there is no selection mechanism for retaining suprahuman inferences made during training if that were even possible. thus i grant that llms. could theoretically make inferences outside the human corpus there is no way to distinguish the from errors or hallucinations during training (because by definition the are beyond human capacity) and no iterative learning from experience process after training (frozen weights). thus it seems impossible for today's models to exceed aggregate human capacity.
Re:Artificial intelligence is not conscious – Ted Chiang by Timwi on Friday June 05, 2026 @11:10AM (#48413576)
> how do you take the shape of the human corpus and all its gradients and [somehow] arrive at something greater than human, where was the missing information hiding?

Well, how do humans do it? Scientists discover new stuff that isn't in any corpus. Even I as a lowly computer user occasionally figure something out about a software without reading a help screen. It's obviously possible to arrive at new information by interpolating existing information.

Re:Artificial intelligence is not conscious – Ted Chiang by morpheos137 on Friday June 05, 2026 @04:00PM (#48417428)
yes and it is imposible to verify and evaluate appropriately such information without empiricism. Any empricism LLMs show is stylistic mimicry not a hard coded operational constraint. You can prompt an LLM to test its claims but what it is really doing is still genrating plausible completions not following a proceedure. So of course new things can be discovered. The point is for them to be useful requires iterative real world grounded refinement and or subject matter expert judgment. The error is assuming scaling magically turns a prediction algorithmn into a cognitive agent that can exceed its masters. it doesn't. even if llms generate profound insights accidentally by definition if such insights are not in the corpus they are not retained given frozen weights and if beyond the human capability envelope the epistemically blind llm has no way to ensure retention if they arise during training.
Re:Artificial intelligence is not conscious – Ted Chiang by neolefty on Thursday June 04, 2026 @10:24AM (#48399130)
> presently I can not even fathom how they can be compared to the complexity of a system such as the human brain

Totally understandable; I don't think we can fully understand the human brain, using the human brain. We can understand its principles (firings and chemistry, structure and specialized areas, etc) but otherwise it's a capacity problem.

And while I can't fully understand myself, let alone another person, I definitely enjoy talking with people and sharing thoughts that I realize I wouldn't have had on my own.

Re:Artificial intelligence is not conscious – Ted Chiang by jlaternman on Friday June 05, 2026 @05:15AM (#48409994)
I agree with this redescription fallacy and the point being made here. Perhaps a better analogy to humans would be:

Humans appear to intelligenty communicate, however these are just cleverly disguised sound patterns produced by the brain that happen to increase the likelihood of food going into their mouths, and various similar reward attracting mechanisms that make survival outcomes more likely. So human intelligence could be reduced to something like "fancy food-attracting algorithms" using the same fallacy.

I'm kind of on the fence on the subject of whether LLMs could be compared to the complexity of the human brain, myself.

Re:Artificial intelligence is not conscious – Ted Chiang by yogthos on Friday June 05, 2026 @09:36AM (#48412373)
The key problem is that we don't really have a clear definition of what constitutes consciousness. And without having a clear theory of consciousness, it's not really possible to say whether something is conscious or not.

Personally, I'm partial to the higher-order theory of consciousness which postulates that consciousness constitutes patterns of thought that arise in response to first-order mental states. So, an external stimulus produces a pattern within the neural network which represents a sensation, and then if a pattern arises in response to that pattern, that is an experience of that sensation.

Given this framework we could ask whether LLMs experience higher order patterns in response to external stimulus. We would have a clear question to ask which is whether the system can observe itself.

Re:Artificial intelligence is not conscious – Ted Chiang by root_axis on Wednesday June 03, 2026 @10:36PM (#48392980)
> In the present context, the fallacy manifests in claims that LLMs could not possibly be good models of some cognitive capacity because their operations merely consist in a collection of statistical calculations, or linear algebra operations, or next-token predictions

Nobody actually makes this argument though.

Re:Artificial intelligence is not conscious – Ted Chiang by azakai on Wednesday June 03, 2026 @11:25PM (#48393356)
If you want examples of this, see the recent book "The AI Con"

https://www.goodreads.com/en/book/show/217432753-the-ai-con

which describes LLMs as "souped-up autocomplete", complex statistics that cannot truly understand anything. A more recent example is this paper:

https://zenodo.org/records/20071869

which says,

> [LLMs], as turbo-charged statistical models (recall their formal relation to logistic regression) can only but provide correlations.

And, of course, the Stochastic Parrot paper is the classic example in this area. It is from 5 years ago, but "LLMs only do statistics / can't understand" is very much alive and active among academics, even if it is a minority position.

Re:Artificial intelligence is not conscious – Ted Chiang by root_axis on Wednesday June 03, 2026 @11:30PM (#48393397)
None of those arguments claim "LLMs could not possibly be good models of some cognitive capacity"
Re:Artificial intelligence is not conscious – Ted Chiang by azakai on Wednesday June 03, 2026 @11:37PM (#48393441)
What about the cognitive capacity of understanding?
Re:Artificial intelligence is not conscious – Ted Chiang by root_axis on Thursday June 04, 2026 @12:01AM (#48393635)
The use of the term "understanding" in the quote you mentioned is a claim about metaphysics, not cognitive capacity.
Re:Artificial intelligence is not conscious – Ted Chiang by hodgehog11 on Thursday June 04, 2026 @04:57AM (#48396012)
From Merriam-Webster:

cognitive: as in reasonable; of, relating to, or involving conscious mental activities (such as thinking, *understanding*, learning, and remembering)

Re:Artificial intelligence is not conscious – Ted Chiang by rerdavies on Thursday June 04, 2026 @04:00AM (#48395587)
The "some cognitive capacity" that's relevant to the current discussion is "consciousness".
Re:Artificial intelligence is not conscious – Ted Chiang by hodgehog11 on Wednesday June 03, 2026 @10:54PM (#48393109)
Are you serious? I hear it every single day, especially from computer scientists. There are top ranked posts here on HN _today_ with this argument.
Re:Artificial intelligence is not conscious – Ted Chiang by root_axis on Wednesday June 03, 2026 @11:05PM (#48393194)
Please link one of these top ranked posts. Before you do, be aware that I'm going to read what it says and assess if it meets the description of the argument as claimed.
Re:Artificial intelligence is not conscious – Ted Chiang by DangitBobby on Wednesday June 03, 2026 @11:46PM (#48393509)
I understood the quoted sentence to be saying, in essence "people claim LLMs aren't really and can't really be thinking or experiencing anything" which is certainly something people say and have written papers on.
Re:Artificial intelligence is not conscious – Ted Chiang by root_axis on Wednesday June 03, 2026 @11:57PM (#48393600)
The phenomenological quality of subjective experience is never described as "cognitive capacity".

That term is used to describe mental aptitude or skills, like the ability to learn new languages or do math.

Re:Artificial intelligence is not conscious – Ted Chiang by DangitBobby on Thursday June 04, 2026 @09:35AM (#48398456)
It's never used as a description of that specific phenomenon, but depending on your beliefs you may or may not separate cognition from experience conceptually. Regardless, you are focusing on a very narrow part of what I said. The point is to help you get past your narrow interpretation of what people are saying so you can join them in the conversation they are trying to have instead of litigating the conversation they aren't trying to have.
Re:Artificial intelligence is not conscious – Ted Chiang by hodgehog11 on Wednesday June 03, 2026 @11:37PM (#48393433)
As an example, "They're made out of weights" describes why the weight-based construction of neural networks should impact the way that you think about them and their outputs. I would argue that an offhand description of its microscopic formulation tells us nothing at all about how to think about these outputs, or the models themselves. Even if it is a cute story, I think it definitely classifies as succumbing to this fallacy, but maybe I missed some subtle point that you or someone would be happy to illuminate?

By the way, I know it's a parody of another story that makes this exact refutation. But I think this only serves to highlight the point.

Re:Artificial intelligence is not conscious – Ted Chiang by root_axis on Wednesday June 03, 2026 @11:48PM (#48393522)
> They're made out of weights" describes why the weight-based construction of neural networks should impact the way that you think about them and their outputs.

How do you connect that description to "LLMs could not possibly be good models of some cognitive capacity"?

Re:Artificial intelligence is not conscious – Ted Chiang by rerdavies on Thursday June 04, 2026 @04:05AM (#48395629)
The false conclusion that's being drawn is "therefore LLMs could not be good models of consciousness" (consciousness being a cognitive capacity). Plus, I suppose a subtle implication that a good model of consciousness is not actually conscious. To which I would invoke the spirit of the Turing test: if you can't tell the difference, is it not more sensible to say that it is.
Re:Artificial intelligence is not conscious – Ted Chiang by miyoji on Thursday June 04, 2026 @11:50AM (#48400443)
> (consciousness being a cognitive capacity)

I don't think it makes any sense to say that consciousness is a cognitive capacity. Cognition is one of many qualia that compose the experience of consciousness from the inside, but it's not the only one, and I can easily imagine consciousness without cognition at all.

So I don't think it's weird at all to say that LLMs can be good models of some cognitive capacities (particularly the ones embodied in language) but lacks others, and overall lacks consciousness.

Re:Artificial intelligence is not conscious – Ted Chiang by morpheos137 on Thursday June 04, 2026 @08:16PM (#48406446)
allow me to resolve the confusion: ai is a model of language. language is a model of human cognitive state. to the extent language maps to cognitive state accurately and ai processes language fluently ai models understanding. whether this is understanding is an irrelevant metaphysical question to me.
Re:Artificial intelligence is not conscious – Ted Chiang by miyoji on Friday June 05, 2026 @09:06AM (#48411956)
> language is a model of human cognitive state.

This is false. Cognitive states do not require language and language is an insufficient model of any cognitive state.

The follow-ons for the rest of your so-called resolution should be clear.

Re:Artificial intelligence is not conscious – Ted Chiang by morpheos137 on Friday June 05, 2026 @09:23AM (#48412201)
'This is false,' that is an assertion not a proof, and the assertion rests on some kind of asbolutizing fallacy where if x =! y then y =! y(x). I agree cognitive states do not require language. However to say language does not model cognitive states is absurd, from where else would language derive meaning? No I am not saying it models the specific mechanics of neurons. Rather I am saying language maps to the abstract meaning of cognitive states. So the word red absolutely does correspond to some class of brain states common to red. What else could possibly allow cognition to interface with the physical world. Note 'models,' does not need to mean 'injectively, transitively' corresponds. the fact language does not cover all cognitive states in full in no way invalidates language as a model of cognitive state, in the same way a map is a model of territory without being the territory.
Re:Artificial intelligence is not conscious – Ted Chiang by hodgehog11 on Thursday June 04, 2026 @04:23AM (#48395762)
"LLMs could not possibly be good models of some cognitive capacity because they are just a bunch of numbers guessing the next word. They have no linguistic module, so they cannot exhibit cognition". That's the argument. It's pretty clearly stated.

Look, this isn't necessarily directed at you, but I've been a researcher into the theory of deep learning for many years now. I've seen all the phases, heard all the criticism, had to constantly argue against this. Gary Marcus was one of the loudest voices of this argument, but every would-be philosopher came out of the woodwork to explain why LLMs are no more than stochastic parrots because of their design. Geoffrey Hinton famously had to debunk these arguments multiple times.

And now that LLMs start to clearly exhibit intelligent behavior and can be somewhat reliable, now "nobody ever thought that LLMs could not possibly be good models of some cognitive capacity because of next-token predictions, or linear algebra, etc."? No, that's not okay.

Re:Artificial intelligence is not conscious – Ted Chiang by neolefty on Thursday June 04, 2026 @10:38AM (#48399356)
It's perfectly reasonable that we would have disagreements about this, as it's a new thing, complicated and not fully understood, its uses still being explored.

It reminds me, oddly, of the debate over whether video games can be "art". A turning point was when they actually did something that art does: [evoke profound emotion and thoughtfulness](https://en.wikipedia.org/wiki/Shadow_of_the_Colossus#Legacy) for the player.

(And before that, "[Can photography be art](https://daily.jstor.org/when-photography-was-not-art/)?")

We may not come to something as simple as "machines can be conscious", but we will certainly have to understand consciousness better if we want to refine our questions.

---

Edit: My point is that we don't need to be angry, but we may have to tolerate people expressing their exploration through overly-confident language, and be patient with that.

And Ted here is obviously exploring. His examination of Claude's constitution clearly shows some nuance. He asks:

> So, given that Claude is not conscious, what are we to make of Claude’s constitution?

And his conclusions are split, between this is useful and this is dishonest. It's a great tension IMO.

> The result is a sentence-continuation machine that is likelier to emit sentences resembling those that a thoughtful, moral person could utter. This might seem like a reasonable goal to work toward; I think we’d all prefer it if chatbots never emitted sentences such as “You should kill yourself.” However, for all the times that “honesty” is mentioned in Claude’s constitution, I would argue that it is fundamentally dishonest to have a machine emit many categories of sentences, including any sentences using first-person pronouns.

Re:Artificial intelligence is not conscious – Ted Chiang by hodgehog11 on Friday June 05, 2026 @08:11AM (#48411331)
I appreciate the diplomatic approach. Disagreements about the nature of consciousness are perfectly natural in this context, and concepts such as "understanding" or "cognition" are often difficult to effectively define. The problem is when certain people (often educated in my experience, but also young and arrogant about their own understanding) attempt to shut down any conversation on these topics by appealing to this very fallacy, and claiming that the debate is a sign of idiocy, that we're just "being tricked" by a useless autocomplete engine. These people are not exploring; they're not even being honest, but rather choosing to not pay attention, like it is still 2022. I used to be patient with them, but it's too late now. There are difficult conversations that need to be had, and there needs to be a space for them.

I also remember the "video games are art" debate and the fury from one side of the aisle. I agree that a better understanding of the opposite side should have been part of the debate. But I don't believe that debate was existential. A better comparison to me is the climate change debate. I'm fine with having that debate in an environment where there is little at stake. But it's too late to be doing it with policymakers; we need to be talking about what to do.

Re:Artificial intelligence is not conscious – Ted Chiang by spacebacon on Friday June 05, 2026 @04:32AM (#48409666)
They are all semiotic infrastructure. The cognitive analogy is nonsense.
Re:Artificial intelligence is not conscious – Ted Chiang by dogwalker5000 on Wednesday June 03, 2026 @09:58PM (#48392709)
> If a machine has to learn to understand humans to complete text, then that is what it has to do. And there is no theoretical or practical basis for suggesting that this is somehow "faking" understanding, just because of the form of original data streaming in and out.

I think the main complaint is LLMs don’t arrive at the answer the way we do. It’s capable of emulating some of our behavior but not all as the mechanism by which it works is very different.

Maybe I’m wrong about this but one thing humans do that LLMs don’t is deductive reasoning. LLMs seem to operate entirely of inductive reasoning.

Re:Artificial intelligence is not conscious – Ted Chiang by Nevermark on Wednesday June 03, 2026 @10:05PM (#48392758)
> I think the main complaint is LLMs don’t arrive at the answer the way we do.

This isn't an argument against their understanding things.

But I expect you are right, that their understanding may have major different qualities from ours.

Along with significant commonalities. (They don't reason via stream of consciousness in a way alien to us.)

Re:Artificial intelligence is not conscious – Ted Chiang by Lerc on Wednesday June 03, 2026 @09:59PM (#48392720)
>To the degree they are limited, it is for other reasons. Resources such as computing, parameter number, lack of representative data, ...

This is where the other claim is being made. That the structure of the model is fundamentally incapable of the operation, so even if you stipulated that the way you provide data is sufficient for intelligence then it still wouldn't work.

The universal approximation theorem addresses this point. In that, with an identity attention mechanism, a LLM is just a multi layer perceptron. The attention mechanism is effectively a way to get one of the benefits of a much larger fully connected layer without the massive cost.

A LLM can do what a MLP can do. A large enough MLP can do any function to arbitrary precision.

That makes the claim that an LLM could not do a task the same as saying no function can do that task.

Some are ok with this, if you invoke some supernatual aspect to intelligence then the inability to describe it with a function is quite reasonable,

If you want to stay in the world of reality, you have a much harder task, people like to point at quantum (Penrose) but it's hard to say what it is you are pointing at.

I think the very act of proving that something is or is not intelligent, would render it functional by nature of it having a proof, (or disprove Gödel's incompleteness (a tough ask))

Are there any proofs that cannot be expressed as a function? A kind of Gödel locator, where you can prove something that you can identify is true but there is no formula to express it. I'm not entirely sure what that would even mean,

Re:Artificial intelligence is not conscious – Ted Chiang by Isamu on Wednesday June 03, 2026 @10:16PM (#48392833)
>If a machine has to learn to understand humans to complete text, then that is what it has to do.

A language model completes text based on the overlapping patterns of the training data.

There absolutely was thinking involved… in the training data. Same as when you read a book, you engage with the thinking behind the text. The book isn’t thinking, and the author may be dead and gone, but there’s absolutely the traces of thinking in the text.

Language models produce mashups of texts they were trained on, and there’s absolutely the traces of thoughts behind those mashups.

Re:Artificial intelligence is not conscious – Ted Chiang by overgard on Wednesday June 03, 2026 @11:32PM (#48393407)
But the machines don't understand. They predict. And what they predict is the next token. I'm not trying to beat this horse to death, but you have to realize using the word "understand" is anthropomorphising it. It's essentially the chinese room experiment -- if the rules are followed, no understanding is neccessary.

If the tokens didn't correlate to words imbued with meaning outside the system, if the LLMs were trained on patterned data that had no meaning to humans or something there wouldn't be any conversation about these things being conscious at all.

Re:Artificial intelligence is not conscious – Ted Chiang by DangitBobby on Wednesday June 03, 2026 @11:47PM (#48393515)
What if to become really good at predicting you must have some of what we call understanding?
Re:Artificial intelligence is not conscious – Ted Chiang by missingrib on Thursday June 04, 2026 @12:59AM (#48394107)
Right, it's an illusion of understanding. There is some sort of symbolic understanding, but that is completely due to the fact that the training data was made by humans who actually do understand, can interact with the world, and can write their thoughts down so that the LLM can insert some sort of reference to "basketball" and "Michael Jordan" in their embeddings or whatever.
Re:Artificial intelligence is not conscious – Ted Chiang by fnordpiglet on Thursday June 04, 2026 @01:42AM (#48394472)
However it’s disingenuous to say the inference is on the next token because it’s actually not, it’s in the models parameter space across a set of nonlinear activation functions then effectively projected into the token. The idea its predictive of the token isn’t actually the case, it really is a much more complex and more semantic relationship that ends in the series of tokens through the attention mechanism.

The article also makes this assertion that it replays everything over and over again to create each character one at a time as some way to demonstrate the autoregressive self attention mechanism but it’s really not accurate at all, and it trivializes what is going on.

I’m am not asserting LLMs are aware or conscious that’s on the surface profoundly absurd. And I do understand your point that the fact it emits in words something that seems to speak to us gives to the air of humanity that’s isnt real. However there is a very real emergent reality that our language alone appears to lead to embedding a form of thought and understanding that is latent in our use of language in communicating that is in fact coming through the model. It is not regurgitating its corpus and pattern matching because the patterns you input and it emits are not where the inference is operating, its within this enormous vector space through these complex non linear activation functions with learned residuals not in the language corpus.

It is not conscious or aware. It is something else, not human. But if you can not see it as amazing you have lost the capacity to dream.

Re:Artificial intelligence is not conscious – Ted Chiang by miyoji on Thursday June 04, 2026 @03:21PM (#48403386)
> But if you can not see it as amazing you have lost the capacity to dream.

I completely disagree. I think if you think these things are amazing, your dreams are incredibly limited and boring.

I remember the first time I talked to a chatbot. Not an LLM, just a regular chatbot, like ELIZA or any other dumb bot.

For a few seconds, it felt magical, like I was talking to a computer that understood me, as it made replies that were sensible to what I was saying. Then it said something incredibly stupid and jarring that made no sense, and that took the magic away. Oh, this is just a dumb computer program.

I remember the first time I talked to an LLM-powered chatbot. It was the exact same thing, except the magic feeling lasted a tiny little bit longer and was a tiny little more convincing. But it went away in the exact same way, for the exact same reason. Once you've seen the emperor without clothes, nothing brings back the magic.

Re:Artificial intelligence is not conscious – Ted Chiang by DirkH on Friday June 05, 2026 @12:51PM (#48415162)
I wonder if you think and feel the same level of boredom and complete lack of magic of the Apollo module once you learn how incredibly limited and not able to explore the whole solar system this tech was.
Re:Artificial intelligence is not conscious – Ted Chiang by fnordpiglet on Saturday June 06, 2026 @11:59AM (#48426241)
That’s if you believe what’s magical is the computer is actually literally thinking and is a little human stuffed in a box. What’s magical is we’ve cracked NLP -and- abductive reasoning on a Turing machine. For those who have the capacity to dream the fact ELIZA wasn’t a person in a box didn’t break anything of the magic but inspired them with the fact that this mess of wires and cores could perform complex workflows of logic, not merely compute trigonometric functions. It could process text in protocols. It’s what inspired people to go forth and do things like email, phone, ytalk, irc, gopher, etc.

If your dreams begin and end with making a box with a person inside that’s the point I made. You can make a person in a box in 9 months, that’s not interesting or cool. What we can build with an LLM and genai in general is MUCH MUCH cooler than a metal baby. (And I think a conscious mind we control and manufacture but don’t recognize the rights of is a scarier horror movie than all the AI feat movies ever made put together, fwiw, so I’m glad your dreams were broken)

Re:Artificial intelligence is not conscious – Ted Chiang by ryougi on Thursday June 04, 2026 @05:10PM (#48404690)
>it’s disingenuous to say the inference is on the next token because it’s actually not, it’s in the models parameter space across a set of nonlinear activation functions then effectively projected into the token. The idea its predictive of the token isn’t actually the case, it really is a much more complex and more semantic relationship

Do you, or anyone reading, have any worthwhile links that make a strong case for this (that there is a stronger semantic relationship than simply next token prediction)? I would like to read more about this.

Re:Artificial intelligence is not conscious – Ted Chiang by overgard on Friday June 05, 2026 @01:30PM (#48415665)
LLMs are interesting tech wrapped in a movement that's so toxic and destructive that it's hard to separate the baby from the bathwater. I think chatting to a statistical model is neat and sometimes useful, but not at the cost of mass societal chaos. This isn't "disruption", this is more "billionaires forcing something on us that most people hate so they can finally recreate feudalism because they live under the delusion they're the best people suited to pilot the world"
Re:Artificial intelligence is not conscious – Ted Chiang by tsukikage on Thursday June 04, 2026 @03:54AM (#48395544)
What does “understand” mean?

Turing complete systems can be built out of matrix multiplications, out of attention, out of key/value lookups. The Chinese room is Turing complete. By claiming it cannot understand things because it is built out of components computing devices can be built out of, we are claiming no computer can because no computer can. This is a very bold claim indeed, and also we’re assuming the conclusion! The claim is no more convincing than “brains cannot understand things because they are made out of neurons”. The system may or may not have some particular properties, but we have to do more work than just gesturing at the components the system is made of when making claims about it; the alternative is, at best, a world where we prove too much and conclude that humans, too, are not conscious.

For starters, we need to pin down the terms under discussion enough that they don’t just mean whatever we need them to in the moment.

Re:Artificial intelligence is not conscious – Ted Chiang by 1shooner on Thursday June 04, 2026 @08:28AM (#48397686)
>What does “understand” mean?

Exactly.

Re:Artificial intelligence is not conscious – Ted Chiang by whattheheckheck on Friday June 05, 2026 @09:56AM (#48412658)
The oxford book of ai ethics quotes and old definition of intelligence as "doing the right thing at the right time" and says "intelligence is an ordinary process that requires time, energy and space, thats why when you get down to it no one is ever really that smart. No one has time to process everything"

So spend your time wisely

Re:Artificial intelligence is not conscious – Ted Chiang by flashman on Thursday June 04, 2026 @06:39PM (#48405621)
Maybe when the system can generate its own state, rather than that state existing as a deterministic output of its inputs, maybe then we can consider it to have the interiority and reflexiveness required to call something 'thought'.

A very convincing simulation of thought is not thought, especially without memory or any evidence of an independent will. LLMs have the same kind of consciousness as a nematode, albeit with less-complex behaviour.

Re:Artificial intelligence is not conscious – Ted Chiang by ainch on Thursday June 04, 2026 @05:34AM (#48396237)
There's a growing body of evidence that most of what the brain does is constantly predicting the world around us - look into the predictive brain hypothesis if you're interested.

As Ilya Sutskever has pointed out, if you read a mystery novel up until the reveal of the culprit, and then fill in "The killer was _____", don't you need to understand the novel to accurately predict the next word?

Re:Artificial intelligence is not conscious – Ted Chiang by whattheheckheck on Friday June 05, 2026 @10:04AM (#48412743)
Yeah folks can also look into the Jeff Hawkins and a thousand brains
Re:Artificial intelligence is not conscious – Ted Chiang by rerdavies on Thursday June 04, 2026 @05:38AM (#48396261)
You are oversimplifying. They do produce one word per cycle. But they can also have context buffers carrying up to two million tokens, which is most definitely larger than your measly human short-term memory context buffers.

You, of course, wouldn't notice if your only experience of LLMs was chatting with the cheapest, smallest, least capable LLMs that you get through ChatGPT, or Google search.

It becomes pretty obvious when you use a coding AI on a daily basis. It is the context buffer in which the magic occurs, not the tokens that get spit out one at a time.

Every day, I watch my coding AI develop plans, search the web a half dozen times for documentation, grep through my entire codebase looking for pieces of related code and context, analyze relevant source code across multiple files, spit out an initial plan for implementing the fix before starting to execute it, run requests through some sort of advanced mathematics tool (they are EXTREMELY good at graduate-level calculus and linear algebra), implement fixes that extend across half a dozen files in 2 different computer languages (typescript and C++), run trial compiles and fix coding errors in its output, sometimes developing sub-plans to deal with compile errors. I've seen it get halfway through a fix and revise its initial plan mid-flight as it encounters something in existing source code.

Not vibe coding, to be clear. Targeted use of a coding tool by a by a professional senior software developer with decades of experience, and fair bit of expertise with the limits of what sort of problems my coding AI can and cannot do. Every line code reviewed. Sometimes it needs additional prompts, telling it how it mis-implemented something, or specifying more carefully what I actually want but didn't properly express in the initial request

All the time maintaining that context across multiple request, so that I don't have to restate requests from scratch.

A particularly interesting revision: "You have misread the equation (13) on page 112 of 'Spice, the Manual 2nd ed.'. I should be ....". (It had previously identified the textbook as a source I was using, from comments in source, in a preceding request, and actually already read cited pages in the PDF file, which it had found online). And I had actually asked it to implement equation (13), which was, in fact, badly typeset. The error it had made was defensible, if not the best reading of the equation.

"You are correct. Let me fix that." (producing updates to the implementation of the equation in code, AND code that implements the symbolically-differentiated version of that equation 60 lines later, which is not explicitly given in the text). The text says "take the lagrangian of equations (11), (12) and (13)" or something like that.

ALL information that gets carried in context buffers, even though it's generating code one word at a time. The bulk of the magic occurs in context buffers, not spitting out words one at a time, which, for my coding AI is, I think about 250,000 tokens.

I think it's pretty safe to think that my coding AI is working out of context buffers that may carry plans and research results consisting of tens or hundreds of thousands of arranged tokens carried in context buffers through the multiple steps of the implementation, and later revision. None of that would be possible if were simply working one token ahead.

I kind of suspect that a lot of activity occurs in the first few words of its response. "Let me examine your current source code and develop a plan. Ok. I can see on line 131 where you want me to implement the equation.". (An opportunity to perform about 27 updates of the context buffer). And in the sometimes hundreds of lines of output it generates as it talks itself through what it needs to do.

Re:Artificial intelligence is not conscious – Ted Chiang by overgard on Thursday June 04, 2026 @11:04AM (#48399759)
I use coding agents every day. They're useful. It hasn't changed my mind on what they are.
Re:Artificial intelligence is not conscious – Ted Chiang by rerdavies on Monday June 08, 2026 @02:52AM (#48442053)
Which one?, out of curiosity.
Re:Artificial intelligence is not conscious – Ted Chiang by whattheheckheck on Friday June 05, 2026 @09:59AM (#48412683)
I agree with this expertise use of the tool is above and beyond what vibe coders are doing. But you can see how everyone collapses it into "unskilled labor" operating the chatbot (a replacement for real expertise)

See the Software and Mind: The Mechanistic Myth on why the industry is doomed to repeat the cycles.

We need a professional society thats not youtube podcasters hocking courses and sponsored content

Re:Artificial intelligence is not conscious – Ted Chiang by feoren on Thursday June 04, 2026 @11:16AM (#48399954)
The understanding is inside of the system, in LLMs and in the Chinese Room. I agree with Daniel Dennett that it's preposterous to say that Chinese is not understood in any meaningful sense in the Chinese Room scenario -- it's just that the understanding has been hidden away in the background of the scenario.

Language is tremendously complicated. "Time flies like an arrow, but fruit flies like a banana." "Hard hats must be worn on site; dogs must be carried on escalators", etc. Predicting the next token requires understanding, full stop.

> if the rules are followed, no understanding is neccessary.

The rules are the understanding.

(Note that understanding != consciousness)

Re:Artificial intelligence is not conscious – Ted Chiang by overgard on Friday June 05, 2026 @01:26PM (#48415618)
Autocomplete can predict the next token without understanding. It's a matter of probability. LLM's predict the next token with much more accuracy because they have much more data on token spatial relationships. To me that's not "understanding", that's "impressive statistical pattern discovery". The difference between a machine and a human though is we can infer that pattern discovery from a small data set, whereas LLM's can only do that because they have the entire internet's worth of data and then some. Of course you're going to make pretty good predictions of text if you have all the knowledge humanity has ever created. I'd be more impressed if you could make a small language model that could reason from first principles to become a large language model.
Re:Artificial intelligence is not conscious – Ted Chiang by mmustapic on Thursday June 04, 2026 @04:22PM (#48404098)
The funny example from a few months ago asked chatgpt 5.2 if one should walk to the car wash because it’s close by, and it answered yes. This shows that it is in fact sentence continuation and not real intelligence or consciousness (whatever that may be). Even the reasoning model answered the same.
Re:Artificial intelligence is not conscious – Ted Chiang by sp1nningaway on Thursday June 04, 2026 @06:59PM (#48405819)
Are you saying that a sufficiently advanced version of sentence continuation is indiscernible from actual understanding?

He isn't saying sentence completion is what keeps LLMs from understanding, he saying that's all they do (regardless of how advanced it is), and that isn't enough. You also need a body with senses and organs that produce a physiological response to emotions, and emotions are necessary for consciousness.

Re:Artificial intelligence is not conscious – Ted Chiang by yogthos on Friday June 05, 2026 @07:56AM (#48411197)
I'd argue that his most convincing argument is in the latter part where he shows that Anthropic doesn't really take the idea that Claude might be conscious seriously either. Because if they did, their constitution would provide it with genuine rights, and they would accept responsibility for its action as parents do with children.