It's not about it being theoretical, it's moreso that the language model is still far more simplistic than our own, and struggles with anything but the most basic relations between nouns. The "horse riding an astronaut" post is a good example of this.[0]

[0] https://garymarcus.substack.com/p/horse-rides-astronaut

See also “Shirt without stripes” https://github.com/elsamuko/Shirt-without-Stripes

It’s funny to see this comment on HN, because it gets refuted so many times by now you should be able to punch into google and find it.

https://twitter.com/Plinz/status/1529013919682994176

I just asked Stable Diffusion to generate 10 images of "horse riding an astronaut" and 10/10 were of an astronaut riding a horse.

This is what it should look like:

https://www.bdaddik.com/en/comics-collectible-postcards/2967...

Modulo s/Lucky Luke/astronaut/g

Note that the image above should be in Dall-E's training set. So it's seen how a horse rides a human. No excuses there.

The comment you're replying to shows Dall-E doing exactly this. Did you click the link?

Wanna guess?

Did you even read the linked article? They even cite the picture you link, it still proves their point though these models have no understanding of the language (like Google claimed).

I think the path forward for something like this is models that learn to execute python code and incorporate the results into their outputs. There are already projects that can generate correct matplotlib calls for prompts like yours, but I don't think we are to the point where those python outputs can be automatically combined with a diffusion model for style or whatever.

One thing they're not very good at is deducing spatial relationships. Concepts like "above", "inside" and "behind", or "after". I'd say the prompts you gave make sense to a human who is thinking of a visual progression from left to right.

I bet you could write a few copilot prompts to generate code which would draw a graph like this, though.

I was surprised while reading your article how good the AIs did. I think it's fascinating that your intuition was that these tools would be able to do a good job of drawing a graph based on a description of it...