A human can learn basic arithmetic, then generalize those principles to bigger number arithmetic, then go from there to algebra, then calculus, then so. Successively building on previously learned concepts in a fully recursive manner. Transformers are limited by the exponential size of their network. So GPT-3 does very well with 2-digit addition and okay with 2-digit multiplication, but can't abstract to 6-digit arithmetic.
DALL-E is an incredible achievement, but doesn't really do anything to change this fact. GPT-3 can have an excellent understanding of a finite sized concept space, yet it's still architecturally limited at building recursive abstractions. So maybe it can understand "green block on a red block". But try to give it something like "a 32x16 checkerboard of green and red blocks surrounded by a gold border frame studded with blue triangles". I guarantee the architecture can't get that exactly correct.
The point is that, in some sense, GPT-3 is a technical dead-end. We've had to exponentially scale up the size of the network (12B parameters) to make the same complexity gains that humans make with linear training. The fact that we've managed to push it this far is an incredible technical achievement, but it's pretty clear that we're still missing something fundamental.
This is false, GPT-3 can do 10-digit addition with ~60% accuracy, with comma separators. Without BPEs it would doubtlessly manage much better.
With multiplication, which requires much more extensive cross-column interaction, accuracy falls off a cliff with anything more than a few digits.
Again, not at all true due to BPEs.
[12] [65] [25] [42] [185]
+ [580] [22] [75] [80] [93]
---------------------------
[706] [75] [300] [278]
> With multiplication, which requires much more extensive cross-column interaction, accuracy falls off a cliff with anything more than a few digits.
You couldn't learn long multiplication if you had to use BPEs, were never told how BPEs worked or corresponded to sane encodings, were basically never shown how to do multiplication, and were forced to do it without any working out.
Quick, what's 542983 * 39486? No writing anything down, you have to output the numbers in order, and a single wrong digit is a fail. (That's easy mode, I won't even bother asking you to do BPEs.)
ML models can learn multiplication, obviously they can learn multiplication, they just can't do it in this absurd adversarial context. GPT-f[1] was doing Metamath proofs on 9-digit division (again, a vastly harder context, they involve ~10k proof steps) with 50-50 accuracy, and we have a toy proof of concept[2] for straight multiplication.
That sounds disappointing but what if instead of trying to teach it to do addition one would teach it to write source code for making addition and other maths operations instead?
Then you can ask it to solve a problem but instead of it giving you the answer it would give you source code for finding the answer.
So for example you ask it “what is the square root of five?” then it responds:
fn main ()
{
println!("{}", 5f64.sqrt());
}But 'understanding' itself needs to be further specified, in order to be tested even.
What strikes me most is the fidelity of those generated images, matching the SOTA from GAN literature with much more variety, without using the GAN objective.
It seems Transformer model might be the best neural construct we have right now, to learn any distribution, assuming more than enough data.
According to your definition of understanding, this program understands something about the concept RED. But the code is just dealing with arbitrary values in memory (e.g. RED = 0xFF0000)
It looks like a variation on plain old image search engine, unreliable at that, as compared to exact matching.
But it has obvious application in design as it can create these interesting combinations of objects & styles. And I loved the snail-harp.
If the AI can actually draw an image of a green block on a red block, and vice versa, then it clearly understands something about the concepts "red", "green", "block", and "on".