Re:Stable Attribution
by l33tman on Monday February 06, 2023 @07:21AM
(#34676311)
"If it does, then that data was indispensable for generating the output for that prompt."
It gets murkier than this, by far. The diffusion models are not just trained on images, but on text. It varies by implementation but Stable Diffusion for example used a pre-trained CLIP transformer network from OpenAI (and subsequently OpenCLIP). CLIP can have internal associations between words that in turn steer the diffusion image generation.
To given a simple example of how this could work (I'm not saying this particular example does work though): the model in total could understand that a "swedish flag" consists of a "yellow cross on a blue background", and it could have yellow crosses in its image training set, but no images of Swedish flags, and it would still understand how to draw a "Swedish flag" based on the language semantics.
Just as a human, actually.
So, this won't work. A style can be inferred and described and it doesn't necessarily need to be in the image training set at all.
It gets murkier than this, by far. The diffusion models are not just trained on images, but on text. It varies by implementation but Stable Diffusion for example used a pre-trained CLIP transformer network from OpenAI (and subsequently OpenCLIP). CLIP can have internal associations between words that in turn steer the diffusion image generation.
To given a simple example of how this could work (I'm not saying this particular example does work though): the model in total could understand that a "swedish flag" consists of a "yellow cross on a blue background", and it could have yellow crosses in its image training set, but no images of Swedish flags, and it would still understand how to draw a "Swedish flag" based on the language semantics.
Just as a human, actually.
So, this won't work. A style can be inferred and described and it doesn't necessarily need to be in the image training set at all.