Re:A preliminary analysis of DALL-E 2 (Marcus, Davis, Aaronson)
by ravi-delia on Monday May 02, 2022 @08:09PM
(#31242739)
I was actually gonna say something like that, but I believe that the full prompt is also made available to the GLIDE portion. Even still, GLIDE isn't a language model it simply can't understand the full relational content of the caption. With changes running from minor tweaks to major upsizes that's probably fixable.
The limitation is caused by the CLIP model they used to encode text and images. It's a separate model only generating an embedding, it's not using attention and pairwise interactions on the whole sequence. This causes Dalle2 to be bad at handling multiple objects with multiple attributes. There is no reason the complete prompt could not be related to the generated image instead of an embed, thus correctly stacking the coloured cubes and assigning the right age to each person mentioned in the prompt.