I don't think that's the right way to think about it.

It's not running an OCR-style algorithm to export data and then passing that data to the LLM for further processing.

Instead, the model itself has been trained on both text and images at the same time.

So the image features end up as a weird ball of floating point numbers mixed together with the floating point numbers for the textual representation of the words in the prompt.

I'd love to understand this more. Strings of text get tokenized into a sequence of integer tokens - is there a similar tokenization step for image inputs? What does that look like?

Ah! Do you think this is like an adjunct or fine-tuning kind of thing or do they retrain GPT 4+ images? That seems like a ginormous task.

When they launched GPT-4 back in March they talked about its image capabilities - I think they trained it into the model from the start, but then spent several extra months on additional safety research, as described in their paper: https://cdn.openai.com/papers/GPTV_System_Card.pdf

I don't think they have trained the model from scratch on images too, but more like Flamingo or similar methods, and the LLM model is at most finetuned but not trained from scratch.

If the model was trained from scratch then I would expect it to be able to develop new emergent behavior like being much better at ascii art, SVG, OpenSCAD scripting at least in the case the model was trained on the usual masked objective but not only on text tokens but image tokens too.

The entire GPT4 model was not retrained to utilize images, instead I think it is much more like show in [1] where an extra layer(s) are learned that take another pretrained image model and map that model to the latent space of the LLM. In [1] this was just a single linear layer. I found it pretty shocking how simple it was. I believe OpenAI is doing something similar.

[1] https://arxiv.org/pdf/2304.10592.pdf

GPT4 was trained to be multimodal from the start.

I wonder if OpenAI themselves even know how the resulting black box actually operates? They know how to train it, sure, but do they understand how it approaches the end results?

I'm pretty confident they don't, and I think they might openly admit that.

There's a lot of ongoing research into interpretability of these models. Anthropic put out a major new paper about that a few days ago: https://www.anthropic.com/index/decomposing-language-models-...