Re:DALL-E 3: Improving image generation with better captions...
by GaggiX on Thursday October 19, 2023 @06:36PM
(#37949727)
They have only distilled the diffusion model used as a replacement of the VAE decoder, not the main diffusion model.
- major improvement is from detailed image captioning
- they trained an image captioning model to produce short and detailed captions
- they use T5 text encoder
- they use GPT-4 to "upsample" short user prompts
- they train a U-net decoder and distill it to 2 denoising steps
- text rendering is still unreliable, they believe the model has hard time mapping word tokens into letters in image