Perhaps I'm very naive in the ways of how these multi-modal LLMs work, I admit not having had time to look into how they're glued together.

I imagined simply tagging the tokens as command and data, ie say instead of just (43) as a token, it's (43, 1) for command and (43, 0) for data. Then ensure through adverserial training that whatever interprets say images does not spit out command tokens.

edit: or better yet, just have it spit ut plain tokens like (43) and ensure they're data tokens by manually converting them to data tokens (43, 0).