Training from video might unleash new capabilities in language models. The volume of data in video format is huge, video contains different information - some common sense patterns that are rarely described in text, it might empower Gato-like RL agents to quickly understand and solve tasks in new environments. Even training from raw audio might open up language models to work for less represented languages / with little written down text, learn to interpret sentiment from tone and improve music composition.

So far we have

- multi modality - text, image, audio, video, math, code. At some point in the future brain scans will also become a cheap modality to train on.

- multi task - finetuning on hundreds of tasks at once and getting zero shot task capabilities

- multi memory - except for the limited buffer, the language models can use retrieval/search over an external corpus or knowledge base and in-batch memories. This opens up the possibility of large scale context, updating recent factual information without retraining; can also make a small LM perform as well as a huge LM without the memory

- multi environment - huge training corpus is great, but is fixed, while environments are dynamic. Agents interacting with games, chat bots, robots, REPL can explore endless scenarios interactively.