I don't have any special insight into the problem, but I'd say whatever form real AGI takes it won't be a language model. Even without AGI these models are massively useful though - a version of GPT-3 that incorporates a knowledge graph similar to TOME would upend a lot of industries.

https://arxiv.org/abs/2110.06176