You are aware the same transformer architecture that underpins LLMs has shown great success in multimodal applications? Such as successfully modeling sound and music?

https://bytedance.github.io/SALMONN/