Re:AnyGPT: Unified Multimodal LLM with Discrete Sequence Mod...
by aaron_wang on Thursday February 22, 2024 @05:06AM
(#39465191)
Totally agree. Another takeaway is that data from different modality has some shared intrinisity, so one can be mapped to another.
Whether data is continuous or discrete, no matter its modality (text, video, music, etc.), we now have an array of proven methods for representing it with discrete tokens, enabling us to use existing sequence modeling architectures (Transformers, linear RNNs).
We live in interesting times!