Tongyi DeepResearch – open-source 30B MoE Model that rivals OpenAI DeepResearch

Re:Tongyi DeepResearch – open-source 30B MoE Model that riva... by idiotsecant on Sunday November 02, 2025 @02:53PM (#45792892)

I think it's the exact opposite - you don't specifically train each 'expert' to be a SME at something. Each of the experts is a generalist but becomes better at portions of tasks in a distributed way. There is no 'best baker', but things evolve toward 'best applier of flour', 'best kneader', etc. I think explicitly domain-trained experts are pretty uncommon in modern schemes.

[ Parent | on HN ]

Re:Tongyi DeepResearch – open-source 30B MoE Model that riva... by viraptor on Sunday November 02, 2025 @03:35PM (#45793193)

That's not entirely correct. Most of moe right now are fully balanced, but there is an idea of a domain expert moe where the training benefits fewer switches. https://arxiv.org/abs/2410.07490

[ Parent | on HN ]

Re:Tongyi DeepResearch – open-source 30B MoE Model that riva... by idiotsecant on Monday November 03, 2025 @08:23AM (#45798699)

Yes, explicitly trained experts were a thing for a little while, but not anymore. Yet another application of the Hard Lesson.

[ Parent | on HN ]