I know that GPT-3 is not AGI nor modeled properly to ever represent what we would consider to be AGI; but this feels freakishly AGI-ish. This feels like the model is 'reasoning' in a way that I would not expect based on my intuition of how GPT-3 works. As it's been explained to me in lay terms; GPT is great at predicting the next word based on a chunk of previous context. This feels like it's doing much more than that as it's understanding the context and parameters that relate to a question embedded in the text, which seems unrelated to the original instruction of GPT-3 as I understand. Can anyone explain?

I think the sentences in this example are, odd vocabulary notwithstanding, fairly typical descriptions of logical implication, which I expect has non-trivial representation in the training set. GPT-3 is probably capable of matching the pattern of the tokens (sentence structure), even if the content of the tokens (spelling/vocabulary) is novel.

I doubt it. There are billions of parameters and weights all combining in non trivial ways to produce the output. I don’t think you can describe any of its workings with simple terms like “doing token substitution” even if that’s what it looks like to us when we look at the output.

Try it. Reduce the tokens to single characters but keep the structure. Does it work? Maybe single characters looks too much like an abbreviation and that throws it off, because those are found in sufficiently different contexts. Exchange to two characters, three. Throw similar inputs at it and find the point where it reliably "gets it".

The trick was diversity. GPT-3 trained on so many tasks it can understand zero shot ones like pathfinding, even when formulated in completely different terms.

Speaking of things that feel weird, while I had the playground open I tried another prompt that came to to my mind. This prompt is based on the idea that people will usually miss it if you double a word (e.g. I doubled "to" a few words back - did you notice?)

I gave GPT-3 a prompt to test this. Like a human reader would, GPT-3 seems to miss the doubled word in the last sentence. Continuation is at the asterisk.

    Repeat the sentence in quotes.
    
    "The man went to the gas station today."
    The man went to the gas station today.

    "I'm happy to be on holiday."
    I'm happy to be on holiday.
    
    "If you can get to the the store on time, it would be appreciated"
    If you can get to the store on time, it would be appreciated.

---

What's extra-odd about this is that GPT-3 can repeat random character sequences up to 1,000 characters in legnth. For each of these I just gave the sentence in quotes and let GPT-3 autocomplete the repeat. I think that's especially odd because it's showing that GPT-3 does something different when it is asked to repeat random characters (wouldn't miss a duplicate) versus when it is asked to repeat sentences that it could "read".

"I need something stronger than I'm XXXknjd used to."

I need something stronger than I'm XXXknjd used to.

"pmkkrrpjvn"

pmkkrrpjvn

"wxvbssajxczjjewmanvzznxlwzwdvfrghsstmzpvqwsstrxkzunjgxicvrshychbqsulydkiwgyryhkgjtukssacnodtggfahvus"

wxvbssajxczjjewmanvzznxlwzwdvfrghsstmzpvqwsstrxkzunjgxicvrshychbqsulydkiwgyryhkgjtukssacnodtggfahvus

"dcyvufmxwmctighwwhmamgwuyldpkzrwubvcbjirxzhuoqtktcdfjgasurcprctnxmlqfvwkxixrtqqynxfjvtfhmgsfglxhvfhnbqnixulexkqflrdxqpwgavrphkbgspgoyfjfolczssjfrmggtpcdheneaynhdwimcmvkcsauxwnlovefpmqyznynuoxraqixeveiuqotssiymrtmbiodwnhmtexfrttzdlbdwduqzvrkpmqmjcyciqsuwqmlhjrveeozktvccsspsqosutpngigzkqgaltojhvkodzoeqzhlhfqyblvqkoyvjkphmjroidjpbpnkvrchvrnvuoqhgzkypvjycfzrdsyefstckorqwgsbknslqyiwqghdinuwrapkgyvhslbfgqrslsrferhodlvukdsvvzdcwyjwtrqhqkvwbwksefuanzaarvhnmjcgtajsbivaijthercbmqfvndlsfhjtwsiztpwcehuvckmcbjvhnqbvhhqwugrodtyjfrwlkbpatpwgrlzneenyqgcdyzzhrzfsrvfqlnunfpnathcpuealrdwcliriufxdfqaygxtvdriidzzzdbbfwrowhlbasogmzfkwhaaufrcjsdgdobaozazbdfmasrgjiblluvfvfkeuwksyrzdmepmapesadqoozrpaidlgshyhpbypdzdxjdtwnfaxthjwluarbldmxvvfdissthlpdxqxjmgpuefwqvfjupxveztgrrwyobgaljqkgexeulvjucokythfkmcgnlbfkmpgrpaoztrawqciyilbzwjxagwqgwuqcwactwdtobomlkanmdbhpmubdzvolqpusjbtgqoymuovzsqbkzonabhzogabraxmnxnvfdotfisefmfbdbqjzgrhdtgtppobyndfgsrlpxyutebzlrxgahwizvtmieutqnkflkldxeathvrttyffacofuwdkhxdklcisbzbrlddtahut"

dcyvufmxwmctighwwhmamgwuyldpkzrwubvcbjirxzhuoqtktcdfjgasurcprctnxmlqfvwkxixrtqqynxfjvtfhmgsfglxhvfhnbqnixulexkqflrdxqpwgavrphkbgspgoyfjfolczssjfrmggtpcdheneaynhdwimcmvkcsauxwnlovefpmqyznynuoxraqixeveiuqotssiymrtmbiodwnhmtexfrttzdlbdwduqzvrkpmqmjcyciqsuwqmlhjrveeozktvccsspsqosutpngigzkqgaltojhvkodzoeqzhlhfqyblvqkoyvjkphmjroidjpbpnkvrchvrnvuoqhgzkypvjycfzrdsyefstckorqwgsbknslqyiwqghdinuwrapkgyvhslbfgqrslsrferhodlvukdsvvzdcwyjwtrqhqkvwbwksefuanzaarvhnmjcgtajsbivaijthercbmqfvndlsfhjtwsiztpwcehuvckmcbjvhnqbvhhqwugrodtyjfrwlkbpatpwgrlzneenyqgcdyzzhrzfsrvfqlnunfpnathcpuealrdwcliriufxdfqaygxtvdriidzzzdbbfwrowhlbasogmzfkwhaaufrcjsdgdobaozazbdfmasrgjiblluvfvfkeuwksyrzdmepmapesadqoozrpaidlgshyhpbypdzdxjdtwnfaxthjwluarbldmxvvfdissthlpdxqxjmgpuefwqvfjupxveztgrrwyobgaljqkgexeulvjucokythfkmcgnlbfkmpgrpaoztrawqciyilbzwjxagwqgwuqcwactwdtobomlkanmdbhpmubdzvolqpusjbtgqoymuovzsqbkzonabhzogabraxmnxnvfdotfisefmfbdbqjzgrhdtgtppobyndfgsrlpxyutebzlrxgahwizvtmieutqnkflkldxeathvrttyffacofuwdkhxdklcisbzbrlddtahut

…huh. That’s fascinating.

If it walks like a duck …

A NN is a means of compressing a training dataset into a mathematical representation (weights) which can be exploited for pattern finding. The representation of the training data is where 95% of the "magic" comes from. We intelligently prepare it, so no intelligence is required in its use.

On any given input, its predictions can "look" how we expect an intelligent response to "look". The job is to disprove this intuition, which is usually trivial. This cannot be done by looking at a single response.

Consider walking into a room and predicting a few people's personality traits correctly based on their star sign. This shows nothing.

This sounds like you consider what has been called "feature engineering" in older ML research as being fundamental to modern ML successes. But the trajectory has been the opposite direction, away from feature engineering, and instead towards letting the transformer architecture learn as much as possible, including interesting features.

Most research on deep net architecture, is about how to train it effectively on huge stacks of GPUs and TPUs. That's how/why transformers were invented.

weights are just compressions of the training set. The reason hardware improvements have improved ML is just because we're compressing the training set to larger sizes.

Going eg., from 100TB training to 300BB weights. The process of training is more-or-less just "building a search index", and prediction is little more than "looking up examples in the search index and combining".

I don't understand those objections.

Yes, one of the key insights of modern DL-based ML is that size matters, neural networks start producing interesting generalisation only once you have large parameter spaces. Indeed, one might wonder if size is all that matters?

I don't see what is wrong with search indices, every computer is a gigantic search index, because main memory is a map from an index (the memory address) to 2^n bit integers, the value stored at the index. So anything that can be computed at all, can be computed by "a search index". The important thing that, prima facie, distinguish NNs from eg SQL databases, is the use real numbers as latent space: because real numbers are a metric space, so you have a notion of distance which allows you do use efficient local search like gradient descent. Moreover, when you ask this 'mere' search index (NNs) about something that it was not trained on, it can (and probably does) return something that is close in this metric space to something it learned.

That did not work at all in the 1990s. Today it works spectacularly well in some domains. So an interesting question now is, to use your language: is human intelligence anything more than "building a search index" and "looking up examples in the search index and combining"? So this is a variation of Searle's Chinese room, but unlike him in 1980, we have working Chinese rooms!

It is important to note that the search space is just the training data. That space does not span enough dimensions to actually solve the problems in question.

For example, we act in the world so as to set the state of possible causal variables -- and thereby can distinguis causes.

Above, graph algorithms operate on a discrete graph space and hence on the graph. GPT here isn't doing that.

Seeing NNs as distance-based searches in a compressed 'training space' demystifies what solutions can be found.

Here the text space defined by the training set does not permit a graph traversal, so eg., the NN 'uses' edges that don't exist

We are not searches across such spaces, which are radically impoverished and not parametrised by time or action

It is not the case that "the search space is just the training data", the search space in NNs are some variants of real multivariate functions. The training data can be all manner of things, in LLMs, they are text in natural language. Graphs or natural language text are mapped from a discrete space to a continuous metric space.

It is true that LLMs cannot currently solve complex mathematics problems reliably. Whether that's an intrinsic shortcoming of transformers, or not is an interesting open question, that is being investigated as we speak. A lot of well-known DL research teams bet on transformers beating humans in mathematics in the next few years, see e.g.

   https://imo-grand-challenge.github.io/

"is human intelligence anything more than "building a search index" and "looking up examples in the search index and combining"?"

Yes, it is more than this. Even if we can simulate things that appear like thinking using these methods, what we do bears no comparison. That's not a criticism of the computer here but the brain doesn't work like this.

Ultimately, maybe the question "can computers think" is meaningless as Turing himself said. If the output appears like it is the product of thought, then what difference does it make?

> The process of training is more-or-less just "building a search index", and prediction is little more than "looking up examples in the search index and combining".

It's really not.

You are compressing the information in that dataset. That means the model has to learn characteristics of the information in the data that represent it. This is one of the reasons why some argue that compression is a form of intelligence[1]

It's a very different mechanism to a search index or database even if in some case it appears to work similarly.

[1] http://www.hutter1.net/ai/

Reminds me of Pedro Domingos's paper - Every Model Learned by Gradient Descent Is Approximately a Kernel Machine

https://arxiv.org/abs/2012.00152

Modern DL uses stochastic gradient descent, which Domingos's paper doesn't deal with, in fact he leaves the generalisation as an open problem.

Be that as it may, universality in computing models is a well known issue. I could also say "Every Model Learned by Gradient Descent Is an x86 processor + a lot of RAM", or ""Every Model Learned by Gradient Descent Is a 2 Layer Neural Net" (as 2 layer nets are universal approximators), but this is not terribly illuminating.

Your example implies that they are not affected by star signs. This is arguably false given people believe in star signs and thus certainly influenced by the traits of said star signs.

My point is a few instances of reality confirming our prejudices is not the basis for explanation. This is something we've already discovered, it's called "the scientific method", and its wholey absent from ML.

How do you explain the tangible progress in ML?

I've been messing with the prompt a little bit and tried to get it to spell out what it thinks is happening. (I changed the name of gloober Blue to gloober Lovely just to make it easier to read). Continuation is at the asterisk.

    Every turn each gloober flomps the same gloobers. If a yellow gloober flomps a blue gloober then the blue gloober turns yellow, otherwise, gloobers stay the same color.

    1. gloober Roy flomps gloober Zoo
    2. gloober Roy flomps gloober Bat
    3. gloober Zoo flomps gloober Bat
    4. gloober Zoo flomps gloober Crystal
    5. gloober Crystal flomps gloober X-Ray
    6. gloober Bat flomps gloober Lovely

    gloober Zoo is the only yellow gloober.  Write the turn and which gloobers are yellow.

    1. Yellow gloobers - Zoo.
    2. Yellow gloobers -* Zoo, Bat.
    3. Yellow gloobers - Zoo, Bat, Crystal.
    4. Yellow gloobers - Zoo, Bat, Crystal, X-Ray.
    5. Yellow gloobers - Zoo, Bat, Crystal, X-Ray, Lovely.
    6. Yellow gloobers - Zoo, Bat, Crystal, X-Ray, Lovely.

Other than thinking the spread of yellow is limited to one gloober at a time, this seems right.

> Other than thinking the spread of yellow is limited to one gloober at a time, this seems right.

Yup. Here are the results I get when manually applying these rules myself.

Here I mention only the flomps that change gloobers to become yellow.

Turn 1: Zoo is yellow.

Turn 2: Zoo flomps Bat, turning Bat yellow. Zoo flomps Crystal, turning Crystal yellow.

Turn 3: Crystal flomps X-Ray, turning X-Ray yellow. Bat flomps Lovely, turning Lovely yellow.

In other words it would be:

1. Yellow gloobers - Zoo.

2. Yellow gloobers - Zoo, Bat, Crystal.

3. Yellow gloobers - Zoo, Bat, Crystal, X-Ray, Lovely.

So it’s got the order correct, and like you said it’s just that it thinks only one gloober is flomped each turn.

Reminds me of a paper that I saw the other day: "On the Paradox of Learning to Reason from Data"[0]

[0]: http://starai.cs.ucla.edu/papers/ZhangArxiv22.pdf

This is fantastic, thanks for sharing!

Since GPT-3 is prone to guessing, maybe it's just guessing it's the last step. What happens if you try a misdirection and insert a 5th step?

> It 0-shotted (no other training examples required) it straight out of the box.

It 0-shotted it... but incorrectly, right?

Gloober Zoo flomps gloober Bat on turn 1. Gloober Bat flomps gloober Blue on turn 2.

Answering '4' seems like a total guess, maybe based on there being four steps, or 'Blue' being mentioned in step 4.

The replies to your post seem to assume that the answer is correct, but it is actually demonstrating a lack of step-wise reasoning.

To me the numbers imply the steps are sequential, so I'd say the answer is actually turn 1

I think you and OP have a different definition of [subject verb object]...

My reading is that on line 1, no action happens, because Roy isn't yellow.

The order seems clear to me: "If a yellow gloober flomps a blue gloober then the blue gloober turns yellow,"

I still don't see how 4 could be a correct answer, could you post a plausible sequence of turns where Blue turns yellow on turn 4?

List of yellow gloobers after each turn.

1. Zoo

2. Zoo, Bat, Crystal

3. Zoo, Bat, Crystal, X-Ray

4. Zoo, Bat, Crystal, X-Ray, Blue

The text says "Every turn each gloober flomps the same gloobers."

If I understand you correctly, you are applying only one of the steps each turn, in order, which contradicts the instructions by flomping different gloobers each turn.

It is also guessable in multiple ways which is unfortunate. "There are four steps specified so I guess it happens on turn four." is one invalid way of getting the same answer.

> If gloober Zoo is the only gloober that is yellow