News for hackers. Stuff that matters.
Re:Transformers struggle with generalizing tasks beyond pre-... by jjtheblunt on Tuesday November 07, 2023 @12:07PM (#38179490)
Don't they essentially memorize all examples they've seen, which contrasts with understanding how to solve problems as in addition with carries on digits etc?
Re:Transformers struggle with generalizing tasks beyond pre-... by zamadatix on Tuesday November 07, 2023 @12:20PM (#38179716)
This oversimplifies too much. GPT4 is able to add random large numbers too large to have been in the data set. When it's wrong it tends to be closely wrong e.g. maybe the first 4 digits of division on a large number by a decimal number with many fractional digits are correct. Neither of these examples can be explained by memorization alone, other operational mechanics are being applied to get these outcomes.
Re:Transformers struggle with generalizing tasks beyond pre-... by jjtheblunt on Wednesday November 08, 2023 @11:57AM (#38193237)
To your point, it would be really neat if we could trace what it's doing to get answers, see why it's sometimes right, sometimes close but wrong!
Re:Transformers struggle with generalizing tasks beyond pre-... by zamadatix on Wednesday November 08, 2023 @02:29PM (#38195656)
You can trace it pretty easily in a local neural net which takes a couple of minutes to train. The only reason we can't trace it in GPT is it's a closed model so we don't have access to do so.

Largely two things come into play: 1) Some part of the neural net is emulating more traditional logic, but it may not always be the most activated part or tuned to be perfect in the answers 2) There isn't really a "jmp" equivalent in a single iteration, so the neural net has to learn to not only do decimal division but do it based on iterating output tokens, continuing perfectly each token output, and choosing to put the right stuff into context and keep that context activated at the right time.

"Activated" in this case means, more or less, the group of neurons specializing on this task are being both fed and listened to.

You can even train a neural net to emulate a traditional addition circuit directly, it's just less efficient than one would think if you're trying to build a general purpose model instead of a specialized one.

Re:Transformers struggle with generalizing tasks beyond pre-... by tsunamifury on Tuesday November 07, 2023 @12:10PM (#38179549)
No, stop repeating this. They use examples to learn and attempt to generalize with on-the-fly modeling.
Re:Transformers struggle with generalizing tasks beyond pre-... by jjtheblunt on Tuesday November 07, 2023 @12:37PM (#38179988)
Citation instead of handwaving that they can learn algorithms? Also, I’ve not asked that before so if it’s being asked frequently, I would imagine it might be because it’s a good question.
Re:Transformers struggle with generalizing tasks beyond pre-... by jjtheblunt on Tuesday November 07, 2023 @12:14PM (#38179617)
Are you claiming the on the fly modeling can learn or reproduce multistep algorithms, like simple addition that the parent comment asked about, and which i asked about above?
Re:Transformers struggle with generalizing tasks beyond pre-... by vidarh on Tuesday November 07, 2023 @01:28PM (#38180779)
I just fed it this:

---

Here is a grammar for a made up language:

    sentence = "sob" subject (verb "ob" object | verb "ad" sentence | "ploj").
    verb = "ab" | "pom"
    object = "foo" | "bar"
Construct a few sentences, and show the sentence so far, and which rule and step of the rule you are currently applying. E.g. sentence= ""; rule=sentence; step="sob".

---

Notice I did not explain the BNF syntax, which is reasonable as it'll have plenty of that in its training data. Notice that I also "forgot" to give it a production for the "subject" rule - leaving it with two choices: It can give up and tell me off, or it can do as best it can and let me know it's had to improvise.

Here's the first sentence construction:

---

First Sentence Construction:

sentence = "";

Rule: sentence; Step: "sob". sentence = "sob";

Rule: subject; Step: (choosing object as a subject for simplicity) "foo". sentence = "sob foo";

Rule: verb; Step: "ab". sentence = "sob foo ab";

Rule: (verb "ob" object | verb "ad" sentence | "ploj"); Step: "ob". sentence = "sob foo ab ob";

Rule: object; Step: "bar". sentence = "sob foo ab ob bar"; (Complete Sentence)

---

It has correctly applied the rules I did give, and opted to "fill in" "object" as a replacement for my missing "subject" rule. There are few enough rules this could be a fluke, but it's a better choice to guess at that substitution than verb or sentence, though hardly takes a genius.

Second:

---

Second Sentence Construction:

sentence = "";

Rule: sentence; Step: "sob". sentence = "sob";

Rule: subject; Step: "bar". sentence = "sob bar";

Rule: verb; Step: "pom". sentence = "sob bar pom";

Rule: (verb "ob" object | verb "ad" sentence | "ploj"); Step: "ad". sentence = "sob bar pom ad";

Rule: sentence; Step: "sob" (this starts a nested sentence). sentence = "sob bar pom ad sob";

Rule: subject; Step: "foo". sentence = "sob bar pom ad sob foo";

Rule: verb; Step: "ab". sentence = "sob bar pom ad sob foo ab";

Rule: (verb "ob" object | verb "ad" sentence | "ploj"); Step: "ob". sentence = "sob bar pom ad sob foo ab ob";

Rule: object; Step: "foo". sentence = "sob bar pom ad sob foo ab ob foo"; (Complete Sentence)

---

This is also correct, and it correctly points out that the arm it picked triggered a nested application of the `sentence` rule.

It stumbled on the third attempt:

---

Third Sentence Construction:

sentence = "";

Rule: sentence; Step: "sob". sentence = "sob";

Rule: subject; Step: "foo". sentence = "sob foo";

Rule: verb; Step: "pom". sentence = "sob foo pom";

Rule: (verb "ob" object | verb "ad" sentence | "ploj"); Step: "ploj". sentence = "sob foo pom ploj"; (Complete Sentence)

---

Note that there is a possible interpretation of the "|" operator which would make its sentence valid, so "technically" it could be, but it would involve (ver "ob" object | verb "ad" sentence | "ploj") to be interpreted as (verb (("ob object)|("ad" sentence)|"ploj")) and that's contrived.

So to answer you: It can, and it does, but not always consistently. One can choose to be disappointed it's not perfect, but for just handing it a grammar and asking it to reason about it with no example text or chances to "practice" whatsoever, I think it's doing quite well.

I've interviewed many software developers who'd struggle with a task like this.

When prodded about the third example, it picks a more reasonable interpretation of the "|", though it's explanation starts out with an error, but then given a correct explanation:

---

Q: How do you figure this? The full rule is "sob" subject (verb "ob" object | verb "ad" sentence | "ploj"). How do you figure the verb is allowed without being followed by either "ad" or "ob"?

A: Upon closer examination, you are correct to point out that the structure following "sob" subject must indeed involve a verb according to the rule provided. The full rule:

"sob" subject (verb "ob" object | verb "ad" sentence | "ploj"). indicates that the sentence must start with "sob", followed by a subject, and then either:

a verb followed by "ob" and an object,

a verb followed by "ad" and another sentence,

or the standalone term "ploj".

---

It goes on to give two correct alternatives to the erroneous one:

---

* "sob foo ploj" — where "ploj" follows directly after the subject without a verb, or * "sob foo ab ob bar" — where "ab" is the verb followed by "ob" and "bar" as the object.

---

While it's not perfect, it can very clearly "learn and reproduce multistep algorithms" to some extent. It also clearly needs to get better at it.

Re:Transformers struggle with generalizing tasks beyond pre-... by tsunamifury on Tuesday November 07, 2023 @09:42PM (#38186115)
Yea I mean you don’t even have to go this far but it’s obvious it can do logical, mathematical and systematic reasoning. I have no idea why people keep insisting it can’t.
Re:Transformers struggle with generalizing tasks beyond pre-... by vidarh on Wednesday November 08, 2023 @05:02AM (#38188470)
Yeah, I asked it that partly because I got curious if it could handle a task that well (because I know plenty of people who can't...), and partly because so many people keep rejecting other reasoning tasks.
Re:Transformers struggle with generalizing tasks beyond pre-... by jjtheblunt on Wednesday November 08, 2023 @12:06PM (#38193417)
People keep asking because they are looking for a rigorous example in a citation that's unambiguous. If it's not obvious, that is even more compelling.

I personally think the more theory trained folks wonder if there will turn out to be some analogy with the corresponding towers of languages and machines recognizing them (regular languages, finite automata, push down automata, turing machines, for examples).

https://a.co/d/8ix6V6m

is an example covering the historical language classes and recognizers for them.

Perhaps LLMs and various architectures will turn out fitting what you describe.

Re:Transformers struggle with generalizing tasks beyond pre-... by GaggiX on Tuesday November 07, 2023 @12:10PM (#38179552)
No they don't.
Re:Transformers struggle with generalizing tasks beyond pre-... by ctoth on Tuesday November 07, 2023 @12:13PM (#38179608)
Yes, actually, they do. See the heading Blessings of scale in [0]

[0]: https://gwern.net/scaling-hypothesis:

Re:Transformers struggle with generalizing tasks beyond pre-... by GaggiX on Tuesday November 07, 2023 @12:23PM (#38179761)
I have trained models on performing additions on numbers with fixed number of digits, just to prove to a friend that a neural network can learn to do addition, I hide in the dataset many combinations of numbers but the model was still able to sum them correctly, therefore it learned to perform addition on numbers with fixed number of digits. So no it's not memorizing, it's something that I have tested myself.
Re:Transformers struggle with generalizing tasks beyond pre-... by jjtheblunt on Tuesday November 07, 2023 @12:27PM (#38179824)
Interesting. Thanks for the comment. What if you ask it to add numbers with more digits?
Re:Transformers struggle with generalizing tasks beyond pre-... by GaggiX on Tuesday November 07, 2023 @12:38PM (#38180005)
The model is trained with a fixed number of tokens, I think it would cause problems with the sinusoidal embeddings layer.
Re:Transformers struggle with generalizing tasks beyond pre-... by Der_Einzige on Tuesday November 07, 2023 @12:46PM (#38180132)
Fine-tuning the model to force it to understand the insane tokenization its forced to use for its vocabulary and perform accurate addition sounds great until you start trying very large numbers or messing with whatever decoding settings you're using (and let me guess, it was beam search).

Or, you can simply ask it to use a tool, like a calculator for you. This is more reliable than fine-tuning a current technique BPE tokenized model ever will be.

Re:Transformers struggle with generalizing tasks beyond pre-... by GaggiX on Tuesday November 07, 2023 @05:56PM (#38184242)
I trained my models from scratch, the tokenization is extremely simple and designed just for this task, decoder-only models can learn to do sums of arbitrarily large numbers if the model is large enough (they are computationally bounded). The decoding method used was greedy or simple sampling.
Re:Transformers struggle with generalizing tasks beyond pre-... by jjtheblunt on Wednesday November 08, 2023 @01:27PM (#38194692)
super cool and interesting link. thanks.