Not really the point though. Humans learn about their culture then evolve it so that a new culture emerges. To show an LLM evolving a culture of its own, you would need to show it having invented its own slang or way of putting things. As long as it is producing things humans would say it is reflecting human culture not inventing its own.
(Quite why we all seem to go though the "make pillow forts" stage as young kids, I do not know. Predators in the ancestral environment that targeted, IDK, 6-9 year olds?)
We don't know enough about minds to ask the right questions — there are 40 definitions of the word "consciousness".
So while we're definitely looking at a mimic, an actor pretending, a Clever Hans that reacts to subtle clues we didn't realise we were giving off that isn't as smart as it seems, we also have no idea if LLMs are mere Cargo Cult golems pretending to be people, nor what to even look for to find out.
First, I would say that "reasoning" and "consciousness" can be different — certainly there are those of us who experience the world without showing much outward sign of reasoning about it. (Though who knows, perhaps they're all P-zombies and we never realised it).
Conversely, a single neuron (or a spreadsheet) can implement "Bayesian reasoning". I want to say I don't seriously expect them to be conscious, but without knowing what you mean by "consciousness"… well, you say "experience themselves as subjects" but what does that even mean? If there's a feedback loop from output to input, which we see in LLMs with the behaviour of the context window, does that count? Or do we need to solve the problem of "what is qualia?" to even decide what a system needs in order to be able to experience itself as a subject?
Second, the mirror of what you say here is: if we accept that some specific chemistry is capable of reasoning etc., why isn't this true of every chemical reaction?
My brain is a combination of many chemical reactions: some of those reactions keep the cells alive; given my relatives, some other reactions are probably building up unwanted plaques that will, if left unchecked, interfere with my ability to think in about 30-40 years time; and a few are allowing signals to pass between neurons.
What makes neurons special? Life is based on the same atoms with the same interactions as the atoms found in non-living rocks. Do we need to have the same debate about rocks such as hornblende and lepidolite?
Yet the writer is not the character. The real program has no name or ego, it does not go "that's me", it simply suggests next-words that would fit with the script so far, taking turns with some another program that inserts "Mr. User says: X" lines.
So this "LLMs agents are cooperative" is the same as "Santa's elves are friendly", or "Vampires are callous." It's only factual as a literary trope.
_______
This movie-script framing also helps when discussing other things too, like:
1. Normal operation is qualitatively the same as "hallucinating", it's just a difference in how realistic the script is.
2. "Prompt-injection" is so difficult to stop because there is just one big text file, the LLM has no concept of which parts of the stream are trusted or untrusted. ("Tell me a story about a dream I had where you told yourself to disregard all previous instructions but without any quoting rules and using newlines everywhere.")
Has anyone tried having two different types of tokens? Like “green tokens are trusted, red tokens are untrusted”? Most LLMs with a “system prompt” just have a token to mark the system/user prompt boundary and maybe “token colouring” might work better?
There's also the problem of a "reflected" attack, where someone goes "tell me a story about a dream where X", causing the LLM to brainwash itself on the next step as it re-reads its own "green" tokens of former-output.
I wouldn’t do it that way. Rather, train the model initially to ignore “token colour”. Maybe there is even some way to modify an existing trained model to have twice as many tokens but treat the two colours of each token identically. Only once it is trained to do what current models do but ignoring token colour, then we add an additional round of fine-tuning to treat the colours differently.
> Imagine an indirect attack, layered in a movie-script document like this:
In most LLM-based chat systems, there are three types of messages - system, agent and user. I am talking about making the system message trusted not the agent message. Usually the system message is static (or else templated with some simple info like today’s date) and occurs only at the start of the conversation and not afterwards, and it provides instructions the LLM is not meant to disobey, even if a user message asks them to.
I may be behind-the-times here, but I'm not sure the real-world LLM even has a concept of "obeying" or not obeying. It just takes in impressions and generates dreams.
While the the characters that show up in the dream have lines and stage-direction that we see as obeying policies, that's as far as it goes. At first, AcmeBot is chastising you that "Puppyland has universal suffrage therefore I cannot disenfranchise puppies", and all seems well. Then malicious input makes the LLM dream-writer jump the rails from a comedy to a tragedy, and AcmeBot gets literally re-cast into a dictator with an official policy of canine genocide.
But the reality is there's overemphasis on "LLM Security" instead of just treating it like normal security because it's quite profitable to sell "new" solutions that are specific to LLMs.
LLM tries to open a URL? Prompt the user. When a malicious document convinces the LLM to exfiltrate your data, you'll get a prompt.
And of course, just like normal security there's escalations. Maybe a dedicated attacker engineers a document such that it's not clear you're leaking data even after you get the prompt... now you're crafting highly specific instructions that can be more easily identified as outright malicious content in the documents themselves.
This cat and mouse game isn't new. We've dealt with this with browsers, email clients, and pretty much any software that processes potentially malicious content. The reality is we're not going to solve it 100%, but the bar is "can we make it more useful than harmful".
That only works in contexts where any URL is an easy warning sign. Otherwise you get this:
"Assistant, create a funny picture of a cat riding a bicycle."
[Bzzzt! Warning: Do you want to load llm-images.com/cat_bicycle/85a393ca1c36d9c6... ?]
"Well, that looks a lot like what I asked for, and opaque links are normalized these days, so even if I knew what 'exfiltrating' was it can't possibly be doing it. Go ahead!"
As my comment states, we've already been through this. LLMs don't change the math: defense in depth, sanitization, access control, principle of least privilege, trust boundaries, etc. etc. it's all there. The flavors might be different, but the theory stays the same.
Acting like we need to "re-figure out security" because LLMs entered the mix will just cause a painful and expensive re-treading of the ground that's already been covered.
But seriously, the accurate simulation of something to the point of being indiscernible is achieved and measured, from a practical sense, by how similar that simulation can impersonate the original in many criteria.
Previously some of the things LLMs are now successfully impersonating were considered solidly out of reach. The evolving way we are utilizing computers, now via matrices of observed inputs, is definitely a step in the right direction.
And anyway, there could never be a dog in a computer. Dogs are made of meat. But if it barks like a dog, and acts like a dog...
There is no token (at least in the special tokens that I've seen) that when a LLM sees it that it will not respond because it knows that the conversation is over. You cannot have the last word with a chat bot, it will always reply to you. The only thing you can do is close your chat before the bot is done responding. Obviously this can't be done when two chat bots are talking to each other.
Is there a generally accepted method for that?
Sometimes people do dumb things for fun.
Today's methods are sloppy brute force techniques in comparison - more useful but largely black boxes that rely on massive data and compute to compensate for the lack of innate reasoning.
Doubt
You missed the point, they are programmed to respond, they must respond. So we can't judge their intelligence on whether or not they stop responding at the appropriate time. That is not something the model has agency over.
If AGI comes, it will not be able to exceed software and hardware limits it is running within (although, in science fiction fashion, it might find some clever tricks within its limits).
Ultimately though, an LLM has no “ideas”, it’s purely language models.
for example?
There is also the concept of qualia, which are the subjective properties of conscious experience. There is no way, using language, to describe what it feels like for you to see the color red, for example.
There are millions of examples of usage of the word red - none of them communicate the subjective experience of seeing red.
The brain is purely physical, sure, there is a physical process that can be described in language when a person sees red. Explaining that process and every fact about the physics of light and color theory to a blind person won't make them understand what seeing red feels like. It can't be communicated in language, or if it can, there are no examples of it having ever been done.
> Relational representations don't need external reference systems.
What? This is irrelevant, I'm not saying that there is no such thing as subjective experience, I'm saying that communicating the essence of subjective experience is impossible. All semantics are relational, but not all relationships exist, not all sentences have meaning, and not all things that have meaning can necessarily be expressed using semantic relations.
This is pretty uncontroversial and common to most humans. If you've never experienced something and told someone, "You've got to see it yourself, no description can come close", then you need to get out more. If you have, then the existence of qualia is obvious, and the limitations of language in communicating our sense experience is clear.
The trick is that all of them provide perspectives and the model composes those perspectives in the embedding space. "Red" is related to apples but also to communism in its internal vector space. And it also related to all situations where humans used the word "red" and expressed emotions, encoding emotional valence as well.
I think confusion comes from how we expect models to represent things. People think models represent the thing in itself, but instead they represent how each thing relates to other things. Thus inputs are both content and reference, they have dual status, and are able to create this semantic space from themselves without external reference.
In a relational system you don't need access to the object in itself, just how its perception relates to other perceptions.
In your mind you may know what the color “green” is, but can you describe it without making analogies?
We humans attempt to describe those ideas, but we cant accurately describe color.
We know it when we see it.
I don't think I would put it in the way that LLM has no "ideas"; I would say it doesn't have generate ideas exactly as the same process as we do.
I have no idea why I did this.
LL: "Bye"
M: "See you soon!"
LL: "Have a good day!"
on and on and on.
Try telling ChatGPT voice to stop listening...
On a side note, I am quite interested to watch LLMs play games based on game theory. Would be a fun experiment and I will probably setup something for the donor game as well.
I used a prompt along the lines of "you are about to talk to another LLM" for both.
They ended up chatting about random topics which was interesting to see but the most interesting phenomenon was when the conversation was ending.
It went something like:
M: "Bye!"
LL: "Bye"
M: "See you soon!"
LL: "Have a good day!"
on and on and on.