Cross-training doesn't always help, I think the architecture is still being worked out. I can't remember the name of it but Google had some multimodal model they released a paper for, maybe six months or a year ago, and it did slightly worse than SOTA ("negative transfer" is the term they used for doing worse from additional modalities), and I recall the paper talking about how reaching near-parity with single mode models was an improvement.

One of their new models, PaLM-E, does show some positive transfer, so it's definitely a potential path for increased capabilities.