I was actually gonna say something like that, but I believe that the full prompt is also made available to the GLIDE portion. Even still, GLIDE isn't a language model it simply can't understand the full relational content of the caption. With changes running from minor tweaks to major upsizes that's probably fixable.