Expressive Text-to-Image Generation with Rich Text

Re:Expressive Text-to-Image Generation with Rich Text by atleastoptimal on Wednesday October 04, 2023 @10:59PM (#37774456)

This is very cool, but it's gimmicky. All of the rich text could simply be a modifier before or after the word (such as an adjective or phrase). Given most LLM work is plain text, this benefit isn't as neatly transferable as prompt engineering.

[ Parent | on HN ]

Re:Expressive Text-to-Image Generation with Rich Text by jbhuang0604 on Wednesday October 04, 2023 @11:41PM (#37774682)

Thanks for the comment! Our method is model-agnostic. It can be easily adapted to any LLM (aka the text-encode) and any text-to-image models.

For example, the method was originally tested in Stable Diffusion 1.4. But we can easily apply it to Stable Diffusion-XL (or any finetuned model like ANIMAGINE-XL) even though the new model has a different text encoder and U-Net weights.

[ Parent | on HN ]

Re:Expressive Text-to-Image Generation with Rich Text by MrNeon on Thursday October 05, 2023 @04:30AM (#37776161)

If you specify the color of an object in a Stable Diffusion prompt the odds of that color appearing somewhere else explode. Tools like Regional Prompter help a lot with this, and I think the 'region specification by prompt' mode works similarly to Rich Text prompting. Not even dall-e 3 is free from color/style leaking to other parts of the image yet.

[ Parent | on HN ]