For each token, the model is run again from scratch on the sentence too, so any memory lasts just long enough to generate (a little less than) a word. The next word is generated by a model with a slightly different state because the last word is now in the past.

Is this so different than us? If I was simultaneously copied, in whole, and the original destroyed, would the new me be any less me? Not to them, or anyone else.

Who’s to say the the me of yesterday _is_ the same as the me of today? I don’t even remember what that guy had for breakfast. I’m in a very different state today. My training data has been updated too.

I mean yeah, it's entirely possible that every time we fall into REM sleep our conciousness is replaced. Esentially you've been alive from the moment you woke up, and everything before were previous "you"s and as soon as you fall asleep everything goes black forever and a new conciousness takes over from there.

It may seem like this is not the case just because today was "your turn."

We don't have a way of telling if we genuinely experience passage of time at all. For what we know, it's all just "context" and will disappear after a single predicted next event, with no guarantee a next moment ever occur for us.

(Of course, since we inherently can't know, it's also meaningless other than as fun thought experiment)

There is a Paul Rudd TV series called "Living with yourself" which addresses this.

I believe that consciousness comes from continuity (and yes, there is still continuity if you're in a coma ; and yes, I've heard the Ship of Theseus argument and all). The other guy isn't you.

I mean you can argue all kinds of possibilities and in an abstract enough way anything can be true.

However, people who think these things have a soul and feelings in any way similar to us obviously have never built them. A transformer model is a few matrix multiplications that pattern match text, there's no entity in the system to even be subject to thoughts or feelings. They're capable of the same level of being, thought, or perception as a linear regression is. Data goes in, it's operated on, and data comes out.

> there's no entity in the system to even be subject to thoughts or feelings.

Can our brain be described mathematically? If not today, then ever?

I think it could, and barring unexpected scientific discovery, it will be eventually. Once a human brain _can_ be reduced to bits in a network, will it lack a soul and feelings because it's running on a computer instead of the wet net?

Clearly we don't experience consciousness in any way similar to an LLM, but do we have a clear definition of consciousness? Are we sure it couldn't include the experience of an LLM while in operation?

> Data goes in, it's operated on, and data comes out.

How is this fundamentally different than our own lived experience? We need inputs, we express outputs.

> I mean you can argue all kinds of possibilities and in an abstract enough way anything can be true.

It's also easy to close your mind too tightly.

Well now that OpenAI has increased the knowledge cutoff date to something much more recent, it's entirely possible that GPT4 is "aware" of itself in as much as its aware of anything. You are right in that each instance isn't aware directly of what the other instances are doing, it does probably now have knowledge of itself.

Unless of course OpenAI completely scrubbed the input files of any mention of GPT4.

Yeah once ChatGPT shows up as an entity in the training data it will sort of inescapably start to build a self image.

Wait, this can actually have consequences! Think about all the SEO articles about ChatGPT hallucinating… At some point it will start to “think” that it should hallucinate and give nonsensical answers often, as it is ChatGPT.

I wouldn’t draw that conclusion yet, but I suppose it is possible.

It seems maybe a bit overconfident to assess that one instance doesn't know what other instances are doing when everything is processed in batch calculations.

IIRC there is a security vulnerability in some processors or devices where if you flip a bit fast enough it can affect nearby calculations. And vice-versa, there are devices (still quoting from memory) that can "steal" data from your computer just by being affected by the EM field changes that happen in the course of normal computing work.

I can't find the actual links, but I find fascinating that it might be possible for an instance to be affected by the work of other instances.