Why do you think next token prediction didn’t take off in multi modal space? That’s right! Because it’s prohibitively slow. A diffusion model, by contrast, can update an entire latent image in parallel at each denoising step. Imagine decoding images or a 10 second video one token/pixel at a time! Secondly, images are not naturally sequential. They are spatial representations of information m. Diffusion can progressively refine both local and global spatial structures in the image.
I actually think there is room for next token multi modal prediction but more where world models are concerned. Like predicting the next set of futures for a robot. But actually now I would opt for decision models instead.
But in short, your analysis, and doubts, are incorrect as to why generalist LLMs are not used instead of bespoke diffusion image gen models.
Regarding whether that's standard to pre-train multi-modal LLM, apparently that is not the case (at least, according to ChatGPT's answer, but that's reasonable - or anyway it was only a wild guess on my side). Apparently it's more common to have a text-only and an image-only pretraining phases, followed by an alignment to make the two embedding spaces a common one.
[1] https://arxiv.org/abs/2409.18869