this hypothesis makes a lot of sense. if indeed gpt-4 is a sparse MoE—which i believe it is—then OpenAI must have tested and proved their initial idea a large capacity MoE LLM model first training/building a smaller one. this smaller model might be gpt-3.5-turbo.