News for hackers. Stuff that matters.
Re:AnyGPT: Unified Multimodal LLM with Discrete Sequence Mod... by ilaksh on Wednesday February 21, 2024 @09:25AM (#39454191)
I don't think it's accurate to call them language models anymore actually. Large Multimodal Model?
Re:AnyGPT: Unified Multimodal LLM with Discrete Sequence Mod... by phh on Wednesday February 21, 2024 @09:38AM (#39454338)
You're right, that calling them language model is no longer accurate. But "Language Model" has a precise signification (which is that it tries to create a perfect representation of the language). Saying "multimodal model", you're not really saying what does it try to represent. "World Model" or "Human Model" would be more accurate at this point
Re:AnyGPT: Unified Multimodal LLM with Discrete Sequence Mod... by t_mann on Wednesday February 21, 2024 @10:02AM (#39454635)
World model sounds too much of physics than it deserves imho. If you want to name it after what's being modeled (which it doesn't need to be, it could also be named after the methods used eg), why not just call it something like general token model?
Re:AnyGPT: Unified Multimodal LLM with Discrete Sequence Mod... by throwanem on Wednesday February 21, 2024 @10:14AM (#39454794)
What they model is meaning. I'd call them "semantic models".
Re:AnyGPT: Unified Multimodal LLM with Discrete Sequence Mod... by Smith42 on Wednesday February 21, 2024 @10:56AM (#39455378)
"Large Observation Model" has a nice ring to it
Re:AnyGPT: Unified Multimodal LLM with Discrete Sequence Mod... by cs702 on Wednesday February 21, 2024 @10:29AM (#39454988)
I'm partial to Large World Model (LWM).

Large Sequence Model (LSM) is OK too.

Re:AnyGPT: Unified Multimodal LLM with Discrete Sequence Mod... by ShamelessC on Wednesday February 21, 2024 @09:40AM (#39454371)
I’m fond of Large Sequence Model, which I’ve seen used a bit recently. If your data can be defined as a sequence of unique symbols, actions, letters, tokens, patches, whatever - a transformer is a great way to model it.
Re:AnyGPT: Unified Multimodal LLM with Discrete Sequence Mod... by kristjansson on Wednesday February 21, 2024 @12:34PM (#39456820)
As long as the underlying representation is a sequence of (embeddings of) tokens , and the training depends on next-token prediction, they should still be called language models. The 'language' part of 'language models' is about the architecture, not the modalit(y/ies).
Re:AnyGPT: Unified Multimodal LLM with Discrete Sequence Mod... by CuriouslyC on Wednesday February 21, 2024 @01:08PM (#39457277)
Is it a "language" model though, if you have a multimodal autoencoder that converts input into concept tokens, and you do sequence extension in concept space?
Re:AnyGPT: Unified Multimodal LLM with Discrete Sequence Mod... by kristjansson on Wednesday February 21, 2024 @01:42PM (#39457709)
Unambiguously, yes.

To elaborate a bit, that's effectively what happens in a natural-language-only LLM. Text tokens are immediately mapped to wide embeddings, which are processed by a transformer, producing another embedding, which is decoded back to a discrete token. To a first approximation, unified multimodal models substitute f: text -> [embedding] with g: image -> [embedding,] and act on a sequence like [*f(t), *g(i), ...]. So... the multimodal model maps text/image parts to wide embeddings, which are processed by a transformer producing another embedding, which is decoded back to a discrete text/image part. Which sounds somewhat similar to the former :).

Of course the function g is a bit more complicated, and decoding takes more work, but the underlying machinery is unchanged. The ability to handle multi-modal inputs/outputs uniformly is exactly what's so exciting about work like this!