Of course it knows what it output a token ago, that's the whole point of attention and the whole basis of the quadratic curse.

It doesn't know anything. It has a bunch of weights that were updated by the previous stuff in the token stream. At least our brains, whatever they do, certainly don't function like that.

To you it might be obvious our brains are different from a network of weights being reconfigured as new information comes in; to me it's not so clear how they differ. And I do not feel I know the meaning of the word "know" clearly enough to establish whether something that can emit fluent text about a topic is somehow excluded from "knowing" about it through its means of construction.

it knows the past tokens because theyre part of the input for predicting the next token. its part of the model architecture that it knows it.

if that isnt knowing, people dont know how to walk, only how to move limbs, and not even that, just a bunch of neurons firing

Similarly, the repair manual cannot reason about novel circumstances, or apply logic to fill in gaps. LLMs quite obviously can - even if you have to reword that sentence slightly.

Holy cow, there's an epidemic of stupid in this thread.

Every time you recall a memory it is modified, every time you verbalise a memory it is modified even more so.

Eye-witness accounts are notoriously unreliable, people who witness the same events can have shockingly differing versions.

Memories are modified when new information, real or fabricated, is added.

It’s entirely possible to convince people to recall events that never occurred.

Which of your memories are you certain are of real occurrences, or memories of dreams?

These are all provable, proven facts.

Who exactly is the subject in this phrase?

If you practice mindfulness meditation, you will come to realize it's not so simple.

But we don't appear to have entirely done that yet. It's just curious to me that the linguistic structure is there while the "intelligence", as you call it, is not.

Not necessarily. You can check this yourself by building a very simple Markov Chain. You can then use the weights generated by feeding it Moby Dick or whatever, and this gap will be way more obvious. Generated sentences will be "grammatically" correct, but semantically often very wrong. Clearly LLMs are way more sophisticated than a home-made Markov Chain, but I think it's helpful to see the probabilities kind of "leak through."

Nobody knows what they are saying either, the brain is just (some form) of a neural net that produces output which we claim as our own. In fact most people go their entire life without noticing this. The words I am typing right now are just as mysterious to me as the words that pop on screen when an LLM is outputting.

I feel confident enough to disregard duelists (people who believe in brain magic), that it only leaves a neural net architecture as the explanation for intelligence, and the only two tools that that neural net can have is deterministic and random processes. The same ingredients that all software/hardware has to work with.

I'm a dualist, but I promise no to duel you :) We might just have some elementary disagreements, then. I feel like I'm pretty confident in my position, but I do know most philosophers generally aren't dualists (though there's been a resurgence since Chalmers).

> the brain is just (some form) of a neural net that produces output

We have no idea how our brain functions, so I think claiming it's "like X" or "like Y" is reaching.

We don't know the architecture or algorithms, but we know it abides by physics and through that know it also abides by computational theory.

Our brains just make words in the same way we catch a tune in our heads.

Then we are culturally conditioned to claim ownership over them and justify them post-hoc (i.e., the ego).

As to what the experience maps to, I think the simplest answer is that our phenomenal experiences are encoded as structures in our brain, but that's not necessary to understanding the difference between words that describe experiences and experiences themselves.

In the case of the brain the encoding is such that various functions "fall out of" it, like being able to relate experiences, etc.

There's no magic proposed here, this is a physicalist functionalist view.

Nothing about this prevents a computer from being sentient. As I said, none of this even matters. The key premise is that LLMs are trained on language and not experiences. Unless you believe that a description of an experience is identical to the phenomenal experience, then we agree on the key premise. Do you think that they are identical?

It's a difficult thing to produce a body of text that conveys a particular meaning, even for simple concepts, especially if you're seeking brevity. The editing process is not in the training set, so we're hoping to replicate it simply by looking at the final output.

How effectively do you suppose model training differentiates between low quality verbiage and high quality prose? I think that itself would be a fascinatingly hard problem that, if we could train a machine to do, would deliver plenty of value simply as a classifier.

If it contains the entire corpus of recorded human knowledge…

And most of everything is shit…

You have no idea what you're talking about. I mean, literally no idea, if you truly believe that.