Wouldn’t this also be correlated somehow in its vector space?

Not to mention that the corpus mostly will have the correct case for most common words like apple. Train it with essays of ESL students and you'll get something else.

From my naïve point of view it seems obvious that at any point where both an „a“ or a „an“ would fit the model randomly selects one of them and by doing so reduces the set of possible nouns to follow.

No, of course not randomly. If you RTA, it should be clear that some prompts are more likely to be followed by an “an” than others.

Yeah, my feeling here was that it's sort of tautological: if GPT predicts "a", then it must then predict a word that would follow "a" and not require "an" (and vice versa). And if you think about it from the opposite direction: if it's working out a response that is eventually going to have "apple" in it, then all the data it's trained on is going to cause it to predict "an" even before it needs to predict "apple".

(Admittedly, all this ML/AI stuff is still beyond my current level of understanding, so I'm sure my thinking here is off.)

I think that's essentially right - if it's already "thinking about" apples, then ' an' will be significantly upregulated at the expense of ' a', then if it does choose to output an ' an' token, ' apple' becomes a highly likely followup.

You could probably test this by seeing if prompts containing a lot of nouns that start with a vowel sound results in output that contains a higher proportion of otherwise unrelated first-vowel nouns. (ie your prompt includes lots of apples, apricots, avocados, asparagus, aubergines, elderberries, eggplants, endives, oranges, olives, okras, onions and you count the proportion of non-food nouns in the result that start with a vowel and non-vowel sound).

The main issue is that GPT is fundamentally an autoregressive language model — it's only predicting the next token based on the prompt at a single time. Every time it wants to predict the next word, it adds the previously predicted word into the prompt, repeating the cycle. We can intuitively guess that the model is 'working out a response that is eventually going to have "apple" in it', but we don't actually know how the model 'thinks' ahead about its response.

To rephrase that for this case: what is the specific mechanism in GPT-2 that (1) makes it realise that the word 'apple' is significant in this prompt, and (2) use that knowledge to push the model to predict 'an'? Finding this neuron would only answer the some portion of (2).

(And to rephrase this for the general case, which gives us the initial question: How does GPT-2 know when, given a suitable context, to predict 'an' over 'a'?)

Do an improv exercise with a friend. Construct a story about a sentient apple one word at a time. I guarantee you that at one point one of you will set up the other with an “an” in your story. This is how I believe this works.

Your friend is thinking ahead that the next word would be apple When they say ‘an’.

But GPT can’t think ahead what token it will add after the one it is on. Or can it? It could “predict” internally the word apple for the next “meaningful” word and output ‘an’ because of this.

Or it could keep outputting articles where appropriate until it outputs an “an” and then output apple having teed itself up.

My friend who is not a native English speaker told me that one thing he struggled the most while learning English was the "a" and "an". He couldn't grasp the concept how it is possible that person knows which determiner to use before saying the word, until he learned it as just the part of the word and then he uses it depending on the context which one can "feel". So when he sees an apple, he says this is an apple.

Exactly what you mean by bigram.

And of course it is not unheard of for a native English speaker to correct themselves if they change their mind about which noun to use. You can imagine being asked to very rapidly verbally classify fruits ("it's an apple, it's an apple, it's a pear, it's an apple,..") that you might well find yourself stumbling like "it's a.. an apple".

This is also why a bunch of words have changed - e.g. apron was originally called napron, but when people were saying "a napron" it got mangled to "an apron"..

Maybe one day "knife" will become "ife".

And narrow from "an arrow"

It's also not unheard of for native English speakers to use it differently (in speech, at least) as a consequence of regional accents. I would use "a history book", because my accent pronounces the h sound. My wife says "an istory book", because her accent drops the h.

But yeah, it should be easier when you're just dealing with text I guess

Does his language have grammatical gender?

The rule is the rule because that’s what make sense phonetically. It’s why you would say “el agua” and “el hacha” in Spanish even though those articles don’t match the gender of those words.

Another interesting case is possessive in French:

  une hache   - ma hache
  une horloge - mon horloge

Now really bake their noodle and ask what they put before the word 'history' or the acronym 'LLM'? :)

Are there languages that don't have similar patterns?

Obviously any romance language includes gender, so you need to use the correct gendered article before the noun.

Japanese has different counting words depending on what you're counting.

I don't know if Mandarin has anything similar.

Mandarin speaker here. There are indeed different counting words for different nouns, so for a bird (鸟) you say 一只鸟 and for a chair (椅子) you say 一把椅子.

I think this would be a valid objection if they stopped there.

But then: “ Testing the neuron on a larger dataset”

If I follow correctly, they test a bunch of different completions that contain “an”. So they are not just detecting the bigram “an apple”, but the common activation among a bunch of “an X” activations where X is completely different.

I don't think you are missing something. I think this whole "GPT-2 is predicting only one word at a time" is a red herring anyway.

Of course it can only answer the next word, because there is only room in its outputs for the next word. But it has to compute much more. It a huge hidden internal state where it has to first encode what the given sentence is about, then predict some general concept in which the continuation goes, decide the locally correct syntactical structure and only from this you can predict the next word.

Author here! I think this is reasonable but I have two responses.

1. It's kinda interesting because this is a clear case where the model must be thinking beyond the next token, whereas in most contexts it's hard to say whether the model thinks ahead at all (although I would guess that it does most of the time).

2. More importantly, the key question here is how it works. We're not surprised that it has this behavior, but we want to understand which exact weights and biases in the network are responsible.

Note also that this is just the introductory sentence and the rest of the article would read exactly the same without it.

> it doesn't seem that surprising that there might be cases where there is a dominant bigram [...] that would trigger an "an" prediction, without actually predicting the following word first

btw I don't really understand what you mean by this. Bigrams can explain the second prediction in a two word pair but not the first.

What happens if you ask it to complete "I climbed up the pear tree and picked a pear. I climbed up the apple tree and picked a"

I asked this to ChatGPT.

Its answer?

"Red apple".

...well played.

I think what the parent was trying to communicate (and what I'm thinking as well) is doubting your premise in 1. ("the model must be thinking beyond the next token").

Rephrase "The model is good at picking the correct article for the word it wants to output next" to "After having picked a specific article, the model is good at picking a follow-up noun that matches the chosen article". Nothing about the second statement seems like an unlikely feat for a model that only predicts one word at a time without any thinking ahead about specific words.

The prompt is:

>I climbed up the pear tree and picked a pear. I climbed up the apple tree and picked

The argument made in the article (IMO an extremely convincing one) is that it wouldn't be able to predict the word 'an' except by observing that the word afterwards must be apple. Otherwise why not pick 'a'?

> It's kinda interesting because this is a clear case where the model must be thinking beyond the next token

I don't see how you get to this conclusion. From all the training data it has seen, "an" is the most probable next word after "I climbed up the tree and picked up". The network does not need to know anything about the apple at this point. Then, the next word is "apple" (with an even higher probability I guess).

I don't understand why your conclusion is that "the model must be thinking beyond the next token": the model doesn't need to do that to generate a well-formed sentence because it's not constrained by the size of the sentence.

As a test, during text generation could you change an “a” to an “an” and see if it changes the noun. Or did it already have a noun in mind and it sticks with that.

Thanks for responding, and for the cool article! Agree this is pretty tangential to the rest of the article.

So (again, I am very much not an expert so please correct me if I'm wrong) I guess my analogy would be if instead of predicting one word at a time, it predicted one letter at a time. At some point, would only be one word that could fit. So if prompted with "SENTE", and it returned "N", that doesn't mean that it's thinking ahead to the "CE" / knows that it is spelling "SENTENCE" already.

Is that a correct way to think of it?

Word shmord, “an apple” is a lexical concept.