News for hackers. Stuff that matters.
Re:Sentence Embeddings have a problem, the reason sometimes ... by carschno on Saturday June 25, 2022 @11:23AM (#31875981)
It's a a pretty big step from "high similarity scores" to concluding that was the reason for failure. High is relative in this context, and the example sentences are arguably more similar to each other than to most other sentences (the definition of similarity I is another topic). From the vector similarity, we don't know much about the actual vector embeddings of the respective sentences, nor about how they impact the model as a whole. Obviously, something goes wrong for the given examples, but I don't think you can pin the problem down to one particular layer in a huge model like Dall-e.
Re:Sentence Embeddings have a problem, the reason sometimes ... by nutanc on Saturday June 25, 2022 @11:51AM (#31876217)
The embeddings provide the guidance for the diffusion models. So they have a direct influence. Similarity of the sentences means that they are around the same area in the multi dimensional space. And when they are used to guide the diffusion models, the images come out wrong. One thing is the experiments I have done are on sentence transformer models. Dalle-2 uses its own sentence embedding afaik.