But a collection of models, with some kind of vector databases(or something more efficient than that), being orchestrated by one or multiple master models.

A large LLM that knows pretty much everything there is to know about the world, seems like a good building block for an AGI, no matter how the rest is built.

LLM's could be like the Broca's and Wernicke's area of an AGI brain. working in unison with dozens of other parts.