News for hackers. Stuff that matters.

A comment on Multi-modal prompt injection image attacks against GPT-4V. Parent comment

Re:Multi-modal prompt injection image attacks against GPT-4V by simonw on Saturday October 14, 2023 @09:57AM (#37880515)
I don't think that's the right way to think about it.

It's not running an OCR-style algorithm to export data and then passing that data to the LLM for further processing.

Instead, the model itself has been trained on both text and images at the same time.

So the image features end up as a weird ball of floating point numbers mixed together with the floating point numbers for the textual representation of the words in the prompt.

I'd love to understand this more. Strings of text get tokenized into a sequence of integer tokens - is there a similar tokenization step for image inputs? What does that look like?

[ Parent | on HN ]
Re:Multi-modal prompt injection image attacks against GPT-4V by jcims on Saturday October 14, 2023 @10:48AM (#37880879)
Ah! Do you think this is like an adjunct or fine-tuning kind of thing or do they retrain GPT 4+ images? That seems like a ginormous task.
[ Parent | on HN ]
Re:Multi-modal prompt injection image attacks against GPT-4V by simonw on Saturday October 14, 2023 @11:03AM (#37881008)
When they launched GPT-4 back in March they talked about its image capabilities - I think they trained it into the model from the start, but then spent several extra months on additional safety research, as described in their paper: https://cdn.openai.com/papers/GPTV_System_Card.pdf
[ Parent | on HN ]
Re:Multi-modal prompt injection image attacks against GPT-4V by GaggiX on Saturday October 14, 2023 @12:38PM (#37881941)
I don't think they have trained the model from scratch on images too, but more like Flamingo or similar methods, and the LLM model is at most finetuned but not trained from scratch.

If the model was trained from scratch then I would expect it to be able to develop new emergent behavior like being much better at ascii art, SVG, OpenSCAD scripting at least in the case the model was trained on the usual masked objective but not only on text tokens but image tokens too.

[ Parent | on HN ]
Re:Multi-modal prompt injection image attacks against GPT-4V by mk_stjames on Saturday October 14, 2023 @01:44PM (#37882544)
The entire GPT4 model was not retrained to utilize images, instead I think it is much more like show in [1] where an extra layer(s) are learned that take another pretrained image model and map that model to the latent space of the LLM. In [1] this was just a single linear layer. I found it pretty shocking how simple it was. I believe OpenAI is doing something similar.

[1] https://arxiv.org/pdf/2304.10592.pdf

[ Parent | on HN ]
Re:Multi-modal prompt injection image attacks against GPT-4V by potatoman22 on Monday October 16, 2023 @11:25AM (#37901141)
GPT4 was trained to be multimodal from the start.
[ Parent | on HN ]
Re:Multi-modal prompt injection image attacks against GPT-4V by jug on Saturday October 14, 2023 @05:37PM (#37884488)
I wonder if OpenAI themselves even know how the resulting black box actually operates? They know how to train it, sure, but do they understand how it approaches the end results?
[ Parent | on HN ]
Re:Multi-modal prompt injection image attacks against GPT-4V by simonw on Saturday October 14, 2023 @05:44PM (#37884528)
I'm pretty confident they don't, and I think they might openly admit that.

There's a lot of ongoing research into interpretability of these models. Anthropic put out a major new paper about that a few days ago: https://www.anthropic.com/index/decomposing-language-models-...

[ Parent | on HN ]