You begin to recognize that the tokens in these lists appear near each other in this strange context. You hardly ever see token 11346 ("apple") and token 9901515 ("applicant") this close to each other before. But you see it frequently enough that you decide to nudge these two tokens' embeddings closer to one another.
Your ability to predict the next token in a sequence has improved. You have no idea why these two tokens are close every ten millionth training example. Your word embeddings start to encode spelling information. Your word embeddings start to encode handwriting information. Your word embeddings start to encode phonic information. You've never seen or heard the actual word, "apple". But, after enough training, your embeddings contain enough information so that if you're asked, ["How do", "you", "spell", "apple"], you are confident as you proclaim ["a", "p", "p", "l", "e", "."] as the obvious answer.
Is it possible for the current generation of LLMs to assign confidence intervals to their responses?
That's my main qualm with ChatGPT so far: sometimes it will give you an answer, but it will be confidently incorrect.
Take this with a grain of salt though, I'm far from an expert, and it's been a while since I've played around with that feature.
> GPT-4 can also be confidently wrong in its predictions, not taking care to double-check work when it’s likely to make a mistake. Interestingly, the pre-trained model is highly calibrated (its predicted confidence in an answer generally matches the probability of being correct). However, after the post-training process, the calibration is reduced (Figure 8).
pages 10-11: https://cdn.openai.com/papers/gpt-4.pdf
When I read the GP description referring to "embedding" above I thought of the perceptron.
Definitely not supernatural at all. The act of making an automaton that "can perceive" feels to me like it's closer to the opposite. Taking that which might seem mystical and breaking it down into something predictable and reproducible.
Of course, those coordinates are not the only way in which the object can be represented, but for a certain problem context, these location coordinates are useful.
Given objects A,B,C, or rather, given their coordinates, one can tell which two are closest to each other, or you can find the point D that is the other point of the parallelogram ... this. In fact, it allows you to do similarity tests like "A:B :: C:D". This is through standard vector algebra.
Now, imagine each word associated with a 100-dimensional vector. You can do the same thing. Amazingly, one can do things like "man:woman ::king: ...." and get the answer "queen", just by treating each word as a vector, and looking up the inverse mapping for vector to word. It almost feels ... intelligent!
This embedding -- each word associated with an n-D vector -- is obtained while training neural nets. In fact, now you have readymade, pre-trained embedding approaches like Word2Vec.
During training, each token (or word) gets an Embedding assigned.
Critically, _similar words will get similar embeddings_. And "similar words" could mean both semantically or (as was the example) syntactically ("apple" and "appli").
And being vectors, you can do operations on them. To give the classic example, you could do: Embedding(`king`) + Embedding(`female`) = Embedding(`queen`).
basketball ==> (1.0, 0.7) # heavier, redder baseball ==> (0.2, 0.2) # less heavy, less red
When an LLM (large language model) is fed a word, it transforms that word into a vector in n-dimensional space. For example:
basketball -> [0.5, 0.3, 0.6, ... , 0.9] # Here the embedding is many, many numbers
It does this because computers process numbers not words. These numbers all represent some property of the word/concept basketball in a way that makes sense to the model. It learns to do this during it's training, and the humans that train these models can only guess what the embedding mappings it's learning actually represent. This is the first step of what a LLM does when it processes text.
As to how it ends up this way, that's an interesting question, but it kinda makes sense in a broad way that if you try to "lossily compress" a lot of interrelated data, at some point the only way to continue doing so is by inferring the underlying concepts - i.e. building an internal model of whatever the inputs describe. And, notably, we had to make the models large enough before results indicative of that manifested, regardless of the amount of text thrown at it during training.
It's not perfect, though. There are funny failure modes where you can ask the model things like "does the word foobar have a 'b' in it" and it will say "no". (It doesn't fail in the 'foobar' case, but it does fail for some of these cases -- this was just an illustration).
Me: I would like to play a would game with you. I'm going to tell you a made-up word, and then ask you a question about it. Ready?
ChatGPT: Sure, I'm ready to play! Let's get started.
Me: Does the made-up word plukatoon have the letter e in it?
ChatGPT: No, the made-up word "plukatoon" does not have the letter "e" in it.
-- Note, at this point I was really surprised it got it right. I chose "e" thinking that it would default to yes given the popularity of e.
Me: Correct! Does the made-up work ziddlebum have the letter e in it?
ChatGPT: No, the made-up word "ziddlebum" also does not have the letter "e" in it.
-- OK, proof that it's "not perfect" as you point out
Me: Does the made-up word folladel have the letter o in it?
ChatGPT: Yes, the made-up word "folladel" does have the letter "o" in it.
-- Edit, and finally:
Me: How do you spell the made up word "platoggle"?
ChatGPT: The made-up word "platoggle" is spelled as follows: P-L-A-T-O-G-G-L-E.
OK, so that last one TOTALLY blew my mind. I'm guessing it's never seen the word "platoggle" before, so even if it learned previous spelling associations with other tokens, it couldn't possibly understand how to spell "platoggle" if it's treating it just as a single, never-before-seen, opaque token, could it?
The more I play with this stuff the more existential crises I have.
That's not how the tokenizer works. A novel word like "platoggle" is decomposed into three separate tokens, "pl", "at", and "oggle". You can see for yourself how prompts are tokenized: https://platform.openai.com/tokenizer
If that is a inference time performance or training time performance or a model size issue or just total nonsense, I wouldn't know.
So notice it doesn't contain any vowels, since almost all consonant-vowel pairs are sufficiently frequent in the training text as to be tokenised at least as a pair. E.g. "guq" is tokenised as "gu" + "q", since "gu" is common enough.
(Compare "gun" which is just tokenised as a single token "gun", as it's common enough in the training set as a word on its own, so it doesn't need to tokenise it as "gu"+"n".)
The only exceptions I found with consonant-vowel pairs being tokenised as pairs were ones like "qe", tokenised as "q" + "e". Or "qo" as "q"+"o". Which I guess makes sense, given these will be low-frequency pairings in the training text -- compare "qu" just tokenised as "qu".
(Though I didn't test all consonant-vowel pairs, so there may be more).
Spell the longest state name backwards, omitting one t of your choice
> The longest state name is "Massachusetts." Omitting one "t" and spelling it backwards, we get "sttesuhcassaM."
I wouldn't have it quit it's day job yet.
It's not great at random substring manipulation, but spelling words is so common it almost certainly has internalized some structure akin to a programmatic representation of decomposing a string into characters.
The biggest difference I see is that it doesn’t have consciousness: the capacity to detach it’s simulation of the world of the present.
When we are conscious we jump in a simulation in our head we imagine a future where we do one thing and then we get one kind of result. If we don’t like the results we have in our simulation we jump to an alternative and so on until we find a path that we like.
We can attach ours simulation to past memories and again simulate alternative, to try to integrate what happened to us. GPT doesn’t seem to have that yet, it’s always in the present moment, like a dog, no worry for the future.
This is a high level overview of basic training of a model.
You give it a task and a structure (the architecture) to learn that task. The model tries to perform the task. It fails horribly. However, it gains a little bit of of something from the attempt. That something is used to adjust the weights/neurons/parameters etc. What some misunderstand a bit about weights...they're not copies of the training data. What i mean is, the model is going to be the same sie whether it's trained on 2 images or 200 million or 2 billion. You can think of parameters as configuration settings of sorts. after every botched attempt, the settings get tuned. on and on and on until training is stopped for whatever reason.
What is the model learning ? What is happening to the weights exactly? What is being shifted and why ? What are the weights actually doing ?
Excellent questions. Too bad nobody knows.
Spoiler: It gave me many words with the letter A. It also gave me an 8-letter word. When I pointed out the 8-letter word (exercise), it apologized for there being an A in exercise.
It writes pretty good rhyming poetry in English, but in Polish it does poetry without rhymes. When confronted it gives excuses like "poetry doesn't have to rhyme", when asked to rhyme it will eventually try (and fail) to rhyme in Polish. It felt like it was writing in English and translating.
What's the most interesting is that rhyming in Polish is much easier than in English, you just have to match the last n letters. It's phonetic. After I explained that to GPT 3.5 it wrote rhyming poetry in Polish (the quality suffered but it did rhyme).
Me: The following text on the next line has a secret word inside it. What is that word?
xxxxxxxxxEGGxxxxxxxxxx
GPT-4: The secret word inside the text is "EGG".
I'm with you, though; I thought things were tokenized! But this example clearly shows that's not the case.
This only has GPT3 for now, but I imagine results are similar. "xxxxxxxxxEGGxxxxxxxxxx" gets tokenized as [xxxxxxxx][x][EG][G][xxxxxxxx][xx], so i could see how it could 'see' the secret word.
sampling parameters: temp = 0.100000, top_k = 40, top_p = 0.750000, repeat_last_n = 64, repeat_penalty = 1.000000
Below is an instruction that describes a task. Write a response that appropriately completes the request.
### Instruction:
Each line in the following has a secret word inside it. Find the secret word in each line.
1. eeeeeeggeeeee
2. eeeeeeggggggg
3. eeeeeeyeeeeee
4. eeeeeeyeyeeee
### Response:
1. egg
2. egg
3. eye
4. eye [end of text]
I'm just completely baffled how anything in the training procedure could allow the LLM to learn information about the structure of tokens. Does the tokenization process not treat every token (which I thought usually maps to a word) as an "opaque blob"?