Just wanted to point out that the notion of a "document" is also an illusion to the LLM. It's processing a sequence of low dimensional spaces into another sequence of low dimensional spaces. The input spaces preserve aspects of content similarity based on co-occurrence. The model learns to transform these spaces into higher order spaces based on the outcome of training.

You couldn't say that the model has a singular sense of self, but it certainly has been trained on data that allows it to mimic it in short spurts, and mimicry is what humans do to learn more complex/abstract tasks. The training goal is not to learn how to "be", but rather to learn how to "do" the parts necessary to continue existing.

"Fake it till you make it" is really all that's required to exist in the world.

For humans, the notion of “self” is also an illusion. We’re just atoms dancing to physics. But that’s not the right level of abstraction from which to do psychology.

Is the notion of an LLM operating on a document useful for building an intuition about how the LLM operates? That’s the more interesting question.

I’d say it is, although it’s not really a document but a stream of tokens. The intuition is that the model is able to build abstractions from that stream. Some of the abstractions are related to goal states of the training metric. However, it’s easy to imagine how some of the goal states pertain to the model’s survival in the “wild”, and part of the abstraction that the model captures for this goal is a reflection of itself.

> In truth, the LLM is an idiot-savant which dreams up "fitting" additions to a given document.

Tbh I'm not too sure that my brain works fundamentally differently. I'm an idiot-savant who responds to stimuli.

Do you know the number of atoms in my coffee cup right now?

If you say "No, I don't" then you're already way ahead of any LLM, even the biggest ones. Remember, I asked about how many are in there right now. The only correct answer is "I don't know", because I haven't even told you whether my mug is full or empty.

Here's copilot's answer:

    how many hydrogen atoms are there in my coffee cup right now
    Copilot said:

    That's an interesting question! The number of hydrogen atoms in your coffee cup depends on the volume of coffee and its composition.

    Let's assume you have a standard coffee cup that holds about 240 milliliters (8 ounces) of coffee. Coffee is mostly water, so we can approximate the number of hydrogen atoms based on the water content.

    Water (H₂O) has 2 hydrogen atoms per molecule. The molar mass of water is approximately 18 grams per mole, and 1 mole of water contains Avogadro's number of molecules, which is about (6.022 \times 10^{23}) molecules.

    1 milliliter of water weighs about 1 gram, so 240 milliliters of water weighs 240 grams. This is approximately ( \frac{240}{18} \approx 13.33 ) moles of water.

    Therefore, the number of water molecules in 240 milliliters of water is: [ 13.33 \times 6.022 \times 10^{23} \approx 8.02 \times 10^{24} \text{ molecules} ]

    Since each water molecule has 2 hydrogen atoms, the total number of hydrogen atoms is: [ 2 \times 8.02 \times 10^{24} \approx 1.60 \times 10^{25} \text{ hydrogen atoms} ]

    So, there are approximately (1.60 \times 10^{25}) hydrogen atoms in a standard coffee cup of 240 milliliters. Keep in mind this is a rough estimate, as the actual number can vary based on the exact composition of your coffee.

If I wanted an explanation of how to approximate it, that's what I would have asked for. You could very probably figure that much out.

No, this is the correct answer. This is the answer I'd be looking for if I was asking that question. That you failed to specify the question is a failure on your part.

Even stupid models understand that if I ask them the population of Denmark they only need to give rough approximation.

no, i expect the system to answer the question I asked. Not the question it thinks I wanted to ask. The question is not underspecified, because the point of it is to demonstrate how the llm will never tell you it doesn't know. Nor will it ever tell you your prompt is underspecified.

> The question is not underspecified, because the point of it is to demonstrate how the llm will never tell you it doesn't know

I don't see how "is not underspecified" follows from the point you were trying to demonstrate. Yes, you wanted it to be well specified, because otherwise the point doesn't work. But you actually failed to generate such a question cause both people and AI interpret it differently.