I'd wonder if that's an artifact of the source data, drilling down in the possibility space to be more like some subset that duplicates the image label- for example pulling tweets with body text and alt text.

Alternatively I guess it could just pull harder towards the prompt, idk.