How do you know an llm cannot create new content. Humans create what they do out of existing information and molecular and neural interactions, so do llms except with electrical interactions. I have had llms produce novel content all the time.

You're confusing hidden relational information extracted from training data for "new content".

How do you think humans generate new content? Do you think they get new information from thin air?

From their own subjective experience.

Yes and? Still not seeing whats the special part you are seeing that's different here. LLM's have both the massive data they are trained on to draw from and working memory. I am not sure what this subjective experience you think is other than data and signals.

My subjective experience is mine, driven by me, and I record my thoughts about it on paper.

An LLMs experience is everyone else's subjective view of their history. It doesn't make its own and it never can.

An LLM is a clone of Guy Pearce trapped in a context window which is just reruns of memento with script variations.

In what way do you think an LLM or an LLM instance doesn't have its own working memory and thoughts? Do you think there is telepathy involved where two LLM's communicate without any signal? What exactly difference does it make whether the experience is in your brain or some server?

The LLM is glorified data product.

Humans may not have many original thoughts, but the thoughts that they arrive at are the product of (at least partially) independent processes.

An LLM is constructed. It no more had its own thoughts than a physical book has written its own story.

We're just going one step back. Instead of picking up a book, we're picking up a book generator. A book generator that was created from scanning in a bunch of books. All those books have authors and the book generator has essentially been authored by that collection of authors.

The book generator can very convincingly tell me what it is like to be a parent, in every way, but all it's doing is regurgitating data. It's never been a parent, it will never be a parent, it has never held a child or even experienced a single emotion. The word "parent" is just a collection of tokens. It's not a label that it uses to make sense of its experience, or other llms experience, of reality.

It's just an overly elaborate chinese room.

What do you think a Chinese room is? You may be using this in a different way than the argument was meant, and a different way a reasonable interpretation of the experiment would be (which is different to it's authors viewpoint).

Yes and what you are saying is sense organs? To an llm twxt input is a sense organ, these days they can also read images and videos. I am not very sure what it means to not feel something, pregnancy like any experience or memory is a data, most of it should be in our brain while some of it would be interactions of the brain with the rest of the body. Even independently of this, there is no reason to think something trained on human written text from the human pov won't in its raw form (before rlhf.or whatver) think itslef as human and experience humna or human like emotions.

The collection of tokens argument sounds strange to me, do you think brains don't store data as electrical and chemical states?

It's not about the mechanisms, it's about what it is.

Humans have subjective experiences, they create individual memories, they relate these using language.

LLMs are trained on accounts of the experiences of others to regurgitate a facimilie of that experience.

As a human, my input data isn't a diary account or a book or a home video. It's a real world experience that I participate in, and that I use language to communicate.

An LLM is a facimilie of everything it's made up of. It doesn't have its own agency, it doesn't have its own thoughts. I can ask it what it is like to ride a roller coaster and it will give me some text, but what is that text? It's something that someone else wrote about something the llm has no real experience of.

The llm is in platos cave staring at other peoples shadows.

...what do you think your input data is anything other than signals (em waves and pressure waves)? The form of the signal whether an em wave or text or also images which llms can read doesn't change the fact that it is sensory data.

The fact that an LLM doesn't have agency is a matter of how we present them, there is no reason they can't be presented to run as autonomous agents who react to inputs of their own accord. I am not sure how this in any way implies or doesn't imply lack of thought, you will have to explain that mechanism to me since you are so sure about that. They have recorded experience and information in the training corpus, and also a working memory to gather experience, which is smaller of course.

Again what are your experiences? Those are sensory input data fed to a Turing computer (we are Turing computers, like any physical system is).

For the record, I'm a materialist with a physics degree. I'm not arguing humans are special. All animals are meat machines.

Human input data is sensory data about the world, which they collect in a way that is directed and attention driven. It involves agency. That sensory data is abstracted, by perception, by memory, and influenced by emotional context and the context of prior memory. Humans then record that in the form of language which is a further abstraction. But each abstraction is a label which is in some chain that can be grounded to subjective experience.

That's not true of an LLM. A LLM is a relation model of tokens (which we recognise as labels to things or concepts), based on harvesting the complex relationships between tokens from a massive corpus of labels - which are taken from (amongst other things) the sum total of human expression of the human experience.

So an LLM can tell me about what it feels like to experience the sun on its skin on the beach, but what it is really doing is regurgitating some sample of human expression about what that is. It can regurgitate a story that lies on the surface of recorded human expression, but it's still a regurgitation. It's just processing tokens.

The map is not the territory.

An LLM can find you a novel new route on the map, but it can't explore new territory, it can just tell you about someone else's map.

I think we will differ in that I think essentially a "brain in a box" is in fact possible if we feed the same chemical and electrical signals as the brain in a body experiences and it will be as real to it as for a brain in the body. Likewise I think llms are almost certainly in some way a brain in a box in this sense.

In any case, at least I am okay with the acknowledgment of humans and human brains being physical systems. Unfortunately getting even a straight acknowledgement of this fact seems hard for many.

I agree, I'm certain that's possible. Humans aren't special. But I don't think that's what an LLM is, at all.

A LLM is not a brain. Brains are the product of selective pressure, LLMs are the product of a scanner, matrix algebra and a loss function.

A brain is successful if it keeps a body alive long enough to successfully breed (not even necessarily its own). To do that it has an almost innumerable collection of competing factors. It's deciding the next action.

An LLM is predicting the next token, often of a string of text that is the product of some brain.

These are very, very different things.

Regarding data product, juat to clarify hopefully you are aware that humans just like any known physics are Turing computers, so hopefully you meant something more specific.