Stability AI releases DeepFloyd IF, a powerful text-to-image model

Re:Stability AI releases DeepFloyd IF, a powerful text-to-im... by danwee on Friday April 28, 2023 @01:27PM (#35744406)

I would be more interested in image-to-text models. Does someone know of any decent model? I saw the GPT4 demo, and they showed that they do image-to-text... but then that was actually a fake (i.e., the model was interpreting the image filename).

[ Parent | on HN ]

Re:Stability AI releases DeepFloyd IF, a powerful text-to-im... by minimaxir on Friday April 28, 2023 @01:40PM (#35744583)

For a fast-but-less-robust model, you can use a ViT encode/GPT-2 decoder model: https://huggingface.co/nlpconnect/vit-gpt2-image-captioning

For a more-robust-but-hard-to-run model, you can use BLIP2: https://huggingface.co/Salesforce/blip2-opt-2.7b

[ Parent | on HN ]

Re:Stability AI releases DeepFloyd IF, a powerful text-to-im... by runnerup on Friday April 28, 2023 @01:37PM (#35744550)

AFAIK, converting an image to a text summary isn't really a thing by itself. The related work would be "visual reasoning" which is the ability to ask things about the image in natural language and get responses back also in natural language.

I believe the current SOTA test for NLVR is VQAv2[0] or GQA[1].

0: https://visualqa.org/ 1: https://arxiv.org/pdf/1902.09506.pdf

[ Parent | on HN ]

Re:Stability AI releases DeepFloyd IF, a powerful text-to-im... by theRealMe on Friday April 28, 2023 @01:37PM (#35744546)

Can you provide a source on gpt4 image model being fake? I haven’t heard that before, though I have wondered why I haven’t heard anything about the image part and don’t have access to image processing myself.

[ Parent | on HN ]

Re:Stability AI releases DeepFloyd IF, a powerful text-to-im... by jkea on Friday April 28, 2023 @01:33PM (#35744495)

MidJourney has the describe function which is kind of like that. Not sure how decent it actually is

[ Parent | on HN ]

Re:Stability AI releases DeepFloyd IF, a powerful text-to-im... by grumbel on Friday April 28, 2023 @01:46PM (#35744658)

CLIP Interrogator[1], which is also build into AUTOMATIC1111, gives quite reasonable results, at least if all you need is a prompt, it can't handle complex interactions:

Image: https://i.imgur.com/husplYZ.png

Output: "a white horse with a sign that says rexel's in space, pixelperfect, inspired by Paul Kelpe, official simpsons movie artwork, alternate album cover, in style of nanospace, by Apelles, pickles, pespective, pop surrealism, ingame, in a space cadet outfit, sifi"

[1] https://huggingface.co/spaces/pharma/CLIP-Interrogator

[ Parent | on HN ]

Re:Stability AI releases DeepFloyd IF, a powerful text-to-im... by jah242 on Friday April 28, 2023 @01:51PM (#35744722)

This is text + image -> text but pretty cool and still might be of interest to you:

https://llava-vl.github.io

[ Parent | on HN ]