But they are self-aware, in fact it's impossible to make a good AI assistant which isn't: it has to know that it's an AI assistant, it has to be aware of its capabilities, limitations, etc.

I guess you're interpreting "self-awareness" in some mythical way, like a soul. But in a trivial sense, they are. Perhaps not to same extent as humans: models do not experience time in a continuous way. But given that it can maintain a dialogue (voice mode, etc), it seems to be phenomenologically equivalent.

Would you say that a calculator is also self-aware, and that it must know its limitations so that it doesn't attempt to do calculations it isn't capable of doing, for instance?

No. The difference is that LLMs can talk, they can use knowledge of themselves in many different ways.

And how do they do that? Be very specific.

Alright. Suppose "meaning" (or "understanding") is something which exists in human head.

It might seem as a complete black box, but we can get some information about it by observing human interactions.

E.g. suppose Алиса does not know English, but she has a bunch of cards with instructions "Turn left", "Bring me an apple", etc. If she shows these cards to Bob and Bob wants to help her, Bob can carry out instructions in a card. If they play this game, the meaning which card induces in Bob's head will be understood by Алиса, thus she will be able to map these cards to meaning in her head.

So there's a way to map meaning which is mediated by language.

Now from math perspective, if we are able to estimate semantic similarity between utterances we might be able to embed them into a latent "semantic" space.

If you accept that the process of LLM training captures some aspects of meaning of the language, you can also see how it leads to some degree of self-awareness. If you believe that meaning cannot be modeled with math then there's no way anyone can convince you.

How does math encode meaning if there is no Alice and Bob? You should quickly realize the absurdity of your argument once you take people out of the equation.

Alright, I wrote an article: https://killerstorm.github.io/2024/12/15/meaning.html

Please let me know which part you find absurd.

It's a good essay but you didn't address my point.

Not sure what you mean... A NN training process can extract semantics from observations. That semantics can be then subsequently applied e.g. to robots. So it doesn't depend on humans beyond production of observations.

The function/mathematics in an NN (neural network) is meaningless unless there is an outside observer to attribute meaning to it. There is no such thing as a meaningful mathematical expression without a conscious observer to give it meaning. Fundamentally there is no objective difference between one instance of a NN with one parameter, f(θ), evaluated on some input, f(θ)(x), and another instance of the same network with a small perturbation of the parameter, f(θ+ε), evaluated on the same input, f(θ+ε)(x), unless a conscious observer perceives the output and attributes meaning to the differences because the arithmetic operations performed by the network are the same in both networks in terms of their objective complexity and energy utilization.

How does the universe encode meaning if there is no Alice and Bob?

One common answer is: it doesn't.

And yet, here we are, creating meaning for ourselves despite being a state of the quantum wave functions for the relevant fermion and boson fields, evolving over time according to a mathematical equation.

(Philosophical question: if the time-evolution of the wave functions couldn't be described by some mathematical equation, what would that imply?)

The universe does not have a finite symbolic description. Whatever meaning you attribute to the symbols has no objective reality beyond how people interpret those symbols. Same is true for the arithmetic performed by neural networks to flash lights on the screen which people interpret as meaningful messages.

> The universe does not have a finite symbolic description

Why do you believe that? Have you mixed up the universe with Gödel's incompleteness theorems?

Your past light cone is finite in current standard models of cosmology, and according to the best available models of quantum mechanics a finite light cone has a finite representation — in a quantised sense, even, with a maximum number of bits, not just a finite number of real-valued dimensions.

Even if the universe outside your past light cone is infinite, that's unobservable.

> Same is true for the arithmetic performed by neural networks to flash lights on the screen which people interpret as meaningful messages.

This statement is fully compatible with the proposition that an artificial neural network itself is capable of attributing meaning in the same way as a biological neural network.

It does not say anything, one way or the other, about what is needed to make a difference between what can and cannot have (or give) meaning.

[flagged]

We've banned this account for repeatedly breaking HN's guidelines.

Please don't create accounts to break HN's rules with.

https://news.ycombinator.com/newsguidelines.html

Must my Search Engine be "self aware" to return images, not text when I do an image search?

I don’t understand what you consider self-awareness to be in this trivial sense. Is Eliza self-aware for example? Eliza maintains a dialogue albeit not obviously as coherent as a modern LLM.

From my perspective, we know what the benchmark is for our own self-awareness, because Decartes gave it to us: Cogito ergo sum. I think therefore I am. We know we think, so we know we exist. That is the root of our self-awareness.

There is a great deal of controversy about the question of whether any of the existing models think at all (and of course that’s the whole point of Turing’s amazing paper[1], and the Chinese room thought experiment[2]) and the best you could say is the burden at the moment is on the people who say models can think to prove that.

Given that, I really don’t see how you can say models have self-awareness at the moment. Models may hypothetically be able to convince themselves they are self-aware via Decartes’ method but notice that Decartes’ proof doesn’t work for us - he was able to pull himself up by his own bootstraps because he knew there was a thought so there must be an “I” who was doing the thinking. We have to observe models from the outside and determine whether or not thought is present, and that’s where the concept behind the Chinese room shows how tricky this is.

[1] Computing Machinery and Intelligence https://academic.oup.com/mind/article/LIX/236/433/986238

[2] https://en.wikipedia.org/wiki/Chinese_room

The only kind of intelligence we know of since childhood is meat-based intelligence. Brain made out of flesh.

So our intuition tells us that flesh is essential. "Chinese room" appeals to this intuition.

But it's a circular argument.

Anyway, I believe they are self-aware, to some extent. Not like a human. But I have no arsenal to convince people who believe intelligence has to be made out of meat.

I certainly don’t think intelligence has to be made out of meat but I do think it’s a complex set of questions and we owe it to ourselves to actually try to tackle the underlying issue of what intelligence and self-awareness etc really are and how we might know whether a model did or did not exhibit these, because we are apt to fool ourselves on both sides of this argument depending on our prior beliefs.

You can see the same thing when people talk about whether animals exhibit self-awareness. There are experiments with dolphins and mirrors for example that definitely suggest that dolphins recognise and might even be amused by their reflection when they see it, but some people find it very hard to reconcile themselves to the idea tgat a dolphin might have a sense of self. I personally find it harder to believe that any particular characteristic would be uniquely human.

Self-awareness is a huge rabbit hole to go down. Its one if the many concepts we think make humans unique, at least in degree, but we never really found a clear way to define or identify it.

I have watched dogs, cats, cows, and chickens pretty extensively. I still couldn't tell you if they are really self-aware, and it ultimately it comes down to a definitional challenge of not having a clear line to draw and identify.

What makes you say LLMs are already self-aware, and how do you define it? And as long as an LLM is functionally a black box, how do you know it comprehends the idea that it is an LLM rather than having simply been trained on that token pattern or given that context as an instruction?

Is it possible to tell intelligence from a simple lookup-table based script by asking questions? I think so.

If you ask one question, there's a chance it was in a lookup table.

If you ask multiple questions from an immense set of questions (like trillions of trillions of trillions..., sampled uniformly), and it answers all correctly, then it's either true intelligence or a lookup table which covers this whole immense space. (I'd argue there's no difference, as a process which makes this nearly-infinite table has to be intelligent.)

Same with self awareness - you can ask questions where it applies...

In either case you are only looking at inputs and outputs. The results may correlate with what you'd consider self-awareness or intelligence, but can you really say that's what you are set ng without understanding how the system works internally.

LLMs are trained on a massive dataset and the resulting model is effectively a compressed representation of that dataset. On the surface you'd have no way of knowing whether the algorithm answering is in fact just a lookup table.

This issue feels very similar to scientific modelling vs controlled studies. Modelling may show correlation, but it will never be able to show causation. Asking a system a bunch of questions and getting the right answers is the same, you're just coming up with a sample set of modelling data and attempting to interpret how the system likely worked only by looking at inputs and outputs.

When I talk with a person I can tell that they are intelligent and self-aware.

Claiming that self-awareness is unknowable concept is inherently unproductive.

People have been using "theory of mind" in practice for millenia so we have to assume it's good for something, otherwise we won't go anywhere. I don't think that knowing internals is important - I don't reach for a scalpel to get what a person means.

When you're talking to a person, though, you also have an understanding of what a human is and what your own experience is.

Its reasonable to interact with another human and expect that they are roughly similar to you, especially when your interactions match what you'd expect.

That doesn't extend as well to other species, let along non-living things that are entirely different from us. They could seem intelligent from the outside but internally function like a lookup table. They also could externally seem like a lookup table while internally matching much better what wed consider intelligence. We don't have context of first hand experience that applies and we don't know what's going on inside the black box.

With all that said, I'm phrasing this way more certain than I mean to. I wouldn't claim to know whether a box is intelligent or not, I'm just trying to point out how hard or impossible it would be today without knowing more about the box.

> Its reasonable to interact with another human and expect that they are roughly similar to you, especially when your interactions match what you'd expect.

It is a default belief that most of us have. The more I learn, the less I think it is true.

Some people have no autobiographical memory, some are aphantasic, others are autistic; motivations can be based on community or individualism, power-seeking, achievements, etc.; some are trapped by fawning into saying yes to things when they want to say no; some are sadistic or masochistic; myself I am unusual for many reasons, including having zero interest in spectator sports and that I will choose to listen to music only rarely.

I have no idea if any AI today (including but not limited to LLMs) are conscious by most of the 40 different meanings of that word, but I do suspect that LLMs are self-aware because when you get two of them talking to each other, they act as if they know they're talking to another thing like themselves.

But that's only "I suspect", not even "I believe" or "I'm confident that", because I am absolutely certain that LLMs are fantastic mimics and thus I may only be seeing a cargo-cult version of self-awareness, a Clever Hans version, something that has the outward appearance but no depth.

> It is a default belief that most of us have. The more I learn, the less I think it is true.

Sure, that's totally reasonable! It all depends on context - I think I'm safe to assume another human is more similar to me than an ant, but that doesn't mean all humans are roughly equivalent in experience. Even more important, then, that we can't assume a machine or an algorithm has developed similar experiences to us simply because they seem to act similarly on the surface.

I'm on the opposite side of the fence as you, I don't think or suspect that any LLMs or ML in general have developed self-awareness. That comes with the same big caveat that its just what I suspect though, and could be totally wrong.

Well, I take this: https://x.com/GrantSlatton/status/1600388425651453953

as evidence that GPT-4 can understand Python, based on assumptions:

1. You cannot execute non-trivial programs without understanding computation/programming language 2. It's extremely unlikely that these kind of programs or outputs are available anywhere on the internet - so at very least GPT-4 was able to adapt extremely complex patterns in a way which nobody can comprehend 3. Nobody explicitly coded this, this capability have arisen from SGD-based training process

That's an interesting one, I'll have to think through that a bit more.

Just first thoughts here, but I don't think (2) is off the table. The model wouldn't necessarily have to have been trained on the exact algorithm and outputs. Forcing the model to work a step at a time and show each step may push the model into a spot where it doesn't comprehend the entire algorithm but it has broken the work down to small enough steps that it looks similar enough to python code it was trained on that it can accurately predict the output.

I'm also assuming here that the person posting it didn't try a number of times before GPT got it right, but they could have cherry picked.

More importantly, though, we still have to assume this output would require python comprehension. We can't inspect the model as it works and don't know what is going on internally, it just appears to be a problem hard enough to require comprehension.

Author of the tweet here. A few notes:

1. No cherry picking

2. This was the original ChatGPT, i.e. the GPT3.5 model, pre-GPT4, pre-turbo, etc

3. This capability was present as early as GPT3, just the base model —- you'd prompt it like "<python program> Program Output:" and it would predict the output

The tendency of your type to immediately accuse anyone who acknowledges that they have subjective awareness of themselves and that this is meaningful with "aha, evidently you believe in magical souls" is very telling, I think.

What looks like “self awareness” is baked in during instruction tuning:

Basically turning any input into a document completion task, giving lots of examples where the completion contains phases like “I am an AI assistant”. This way, if GPT-3 would have completed your question with more questions that are similar, “assistants” will complete it with an answer, and one that sounds like it was spoken by someone who claims to be an AI assistant.

He is just giving the audience what it wants to hear.

There is a huge collective denial of how bad progress is stalling out because the economy and markets have basically shoved all in on AGI.

There is really no option here but to keep the poker face and bluff going , hoping to not get called as long as possible.