Yea, after all these LLMs are predicting one sequence of tokens from another sequence of tokens and the tokens could be anything, it just "happens" that text has the most knowledge and the easiest to input, then there are image, sound, video, but tokens could also be learned from world experience in RL:

Transformers are Sample-Efficient World Models:

https://github.com/eloialonso/iris#transformers-are-sample-e...