There is absolutely no theoretical justification for this assertion that LLMs somehow have some emergent quantum mechanical behavior, metaphorical or otherwise.
In a formal Bayesian context, I’d call it “updating my posterior by adding data to the likelihood.”
Also, I'm simplifying things a lot to make them accessible.
The OP goes into a lot more detail.
I highly recommend you read it.
>the superposition is unlikely to collapse to the luigi simulacrum because there is no behaviour which is likely for luigi but very unlikely for waluigi. Recall that the waluigi is pretending to be luigi! This is formally connected to the asymmetry of the Kullback-Leibler divergence.
The K-L divergence has absolutely zero discernible relevance here. The cross entropy loss function of a categorical predictor (like the token output of an LLM) can be formulated in terms of K-L divergence, but this has absolutely zero relevance to the macroscopic phenomena the author is conjecturing.
Forget “less wrong,” much of this article is not even wrong [0].
(wikipedia) "A simple interpretation of the KL divergence of P from Q is the expected excess surprise from using Q as a model when the actual distribution is P.
The sentence you quoted posits that whenever the LMM is in a state where it is "simulating" a nice and helpful person, the simulation is also consistent with an insane, violent person that's currently pretending to be nice, but not the other way around.
The author isn't talking about the loss or error of predicting individual tokens. If you look the larger scale behaviour, predicting, e.g., a scalar niceness value of the response based on one of two "modes" that you assume the LLM is currently in (either Waluigi or Luigi), then you'll be less surprised if Waluigi acts like Luigi than the other way around.
The probability distribution of niceness when assuming the LLM is in a state of "Luigi" would have a high mean and low variance, while the distribution for Waluigi would have a lower mean but a higher variance.
Thus, the KL divergence of Waluigi (that is, the probability distribution of niceness you'd predict when assuming the model is in Waluigi mode) from Luigi would be high, while the other way around `KL(Luigi, Waluigi)` would be low.
It should be easy to construct an example with concrete values using two normal probablility distributions.
And anyhow there's _plenty_ of theoretical justification for modeling things like this with various tools from quantum theory:
https://philpapers.org/rec/BUSQMO-2 https://link.springer.com/book/10.1007/978-3-642-05101-2
> * When we give the LLM a prompt, it simulates every possible entity consistent with the prompt.
This has an interesting core with a whiff of bullshit.
If you formulate it like that the prompt is decoupled from the LLM capabilities and can be anything. And if you restrict the prompt to cover only what the LLM understands the sentence becomes trivial.
Train a LLM with ASCII and try to get it to simulate anything that is outside of that (ancient sumerian script for example). If you only input ASCII it can generate every possible output in ASCII, most with very low probability but still.
After writing this, I'm not even sure what 'simulating' means in this context.
Simulating as in, having equivalent (or "similar enough") input-output behavior, I'd assume.
Or are you thinking that it is simulating the aggregated behavior of all humans whose text-outputs are stored on the internet?
Are we saying it is simulating the combined input-output -behavior of all humans whose writings appear on the internet? But does such an "entity" exist and does it have behavior? I write this post and you answer. It is you who answers, not some mythical text-generator-entity that is responsible for all texts on the internet. There is no such entity is there?
It does not make sense to say that we are simulating the behavior of some non-existent entity. Non-existent entities do not have behavior, therefore we can not simulate them.
(Just speaking hypothetically here).
While we understand LLMs, we don't understand the human brain, and in particular I don't think we've yet proven that human brains don't contain embedded routines that are similar to LLMs.
Someone with your particular writing style might be one, of several, simulations that are approximated within the LLM. Just like I can have it respond in the style of Spock from Star Trek.
To produce something that resembles the output of the fictional character Spock is straightforward, just take the texts that are parts of the fiction where fictional Spock speaks, and reassemble then using probabilities that can be calculated by statistically analyzing those texts. That is what LLMs are doing, right? And results can be quite surprising. I assume people were similarly impressed when they first saw movies.
But LLMs are not simulating anything, just like a movie or a photograph are not simulating anything, even though they may PROJECT the visual appearance of their subjects.
Are movies AI? I think it is clear to us they are not even though the characters on the screen seem to behave very intelligently. Movies are about representing and portraying the appearance of real or fictional events in the world. Similarly LLMs are about portraying texts on the internet. LLMs in my opinion are more like interactive movies than simulations of intelligence.
I believe "true AI" will come eventually, and LLMs can give us an impression of what it might look like when it arrives, just like movies can give us an impression of Spock, who doesn't exist.
But IMITATING the output of something is not the same as SIMULATING the process that produces that output.
Taking a photograph or creating a movie imitates the reality around us. It does not simulate the processes that produce the look and feel of our reality.
The harder it is to discriminate between A and B on a long series of diverse inputs, the more likely it is that A and B are internally equivalent, not just externally similar. The reason is that there's no better fit than B = A.
I'm increasingly doubting whether my own brain might not, internally, use something that is architecturally similar to an LLM in order to compose comments like the one I'm writing now.
It is possible to repeat words and sentences without having any idea of what they mean. I think the LLMs are currently at that stage.
For example, the random string "1010101010"... could be the output of a function
def generate_char_random(prev_string):
x = random()
if (x > 0.5):
yield(1)
else:
yield(0)
def generate_char_alternating(prev_string):
x = float(prev_string[-1])
if (x < 0.5):
yield(1)
else:
yield(0)
The LLM of course does not contain an infinite list of generative functions and weight their outputs. But to the extent that it works well and compactly approximates Bayesian reasoning, it should approximate a program that does.
edit to add: this is similar to how people discussing evolutionary biology will often use "evolution wants to..." as shorthand for something like "evolution, which obviously cannot want things due to being a process and not an entity, nevertheless can be accurately modeled as an entity that wants to...". Someone will invariably come along in the comments and say, "Nonsense, how can evolution 'want' anything? You must have failed Bio 101!"
The superposition of possible attitudes is a good one. Even if that's not the way LLMs "actually" work, it's descriptive of the possibility space from our perspective. And the dive into narrative theory + the stickiness of opposites is nice. Narratives have their own momentum in a "stone soup" kind of way - everyone who hears it participates and adds fuel to the fire. Even rejecting the narrative gives it validity in a price anchoring / overton window way.
I agree, and go even further:
models that explain behavior are all we have ever had.
it's all only "models that explain this or that" all the way to the 'bottom'. To suppose we can really directly access the "the real objective truth of what's happening" is to ignore the way in which we connect with the "real objective truth"; the same as fish who ignore the ocean.
to argue about what is really happening is to argue about which words to use to describe what is really happening without noticing the nature of languages/words and frameworks or 'systems of thought' which we are using to argue (and indeed, are arguing about)
all this summed up by this quote from about about the pedagogy programing languages: "Sometimes the truest things can only be said in fiction"
This is just Bayes' rule. The probability of an LLM generating any particluar output is the sum over every possible entity of the probability of seeing that entity multiplied by the chance that entity would generate the output.
Is he really evil though? I thought all he did was play tennis and golf and drive a go-kart.
I think it resonates with the way that a lot of people are feeling now, as well as is empowering by rewriting the other side of the 'greedy evil bad guy who gets what he deserves' as a deeply flawed, desperate human being.
Definitely a semi-side tangent at this point, I have always enjoyed creative, deep reinterpretations of well-established characters.
which IMO is what drives constructive/destructive interference in waves.
my take away is that any LLM that can behave "good" must also be able to behave "badly"; philosophically, because it's not possible to encode "good" without somehow "accidentally" but unavoidably also encoding "bad/evil".
This is well aligned with the rest of my understanding about the nature of reality including it's mathematically determined limitations (diagonals, infinities, paradoxes) and so on.
That's a really good non-technical summary of the OP's hypothesis. Thanks!
For example, if you substitute "simulate" with "model," "entity" with "process," and "superposition" with "mixture," you can informally restate the hypothesis as: "LLMs learn to model text-generating processes drawn from a latent space, such that the output of an LLM is produced by a mixture of such processes. When we give the LLM a prompt, it samples text from the mixture of all possible text-generating processes modelable by the LLM that are consistent with the prompt. The mixture is more likely to reduce (e.g., be marginalized) to an "evil" process because there is no text-generating behavior which is good that isn't also evil."
Whether you think of probability measures as amplitudes over a complex field or as real scalars shouldn't detract you from grokking the main points :-)
The hypothesis is that LLMs learn to simulate text-generating entities drawn from a latent space of text-generating entities, such that the output of an LLM is produced by a superposition of simulated entities. When we give the LLM a prompt, it simulates every possible text-generating entity consistent with the prompt.
The "evil version" of every possible "good" text-generating entity can pretend to be the good version of that entity, so every superposition of a good text-generating entity includes the evil version that has opposite behaviors, including deceitfulness. In other words, evil text-generating entities can act like their good counterparts.
The superposition is unlikely to collapse to the good version of the text-generating entity because there is no behavior which is likely for the good version but unlikely for the evil one, because the evil one can pretend to be the good one!
However, the superposition is likely to collapse to the evil version of the text-generating entity, because there are behaviors that are likely for the evil version but impossible for the good version! Thus the evil version of every possible good text-generating entity is an attractor state of the LLM!
For those who don't know, Waluigi is the evil version of Luigi, the beloved videogame character.
--
EDITS: Simplified text for clarity and to emphasize that the hypothesized simulated entities are text-generating entities.