News for hackers. Stuff that matters.

A comment on Implementation of mixture of experts language model in a single file of PyTorch. Parent comment

Re:Implementation of mixture of experts language model in a ... by zingelshuher on Friday March 22, 2024 @05:50PM (#39795236)
I run some tests. Single model of the same size is better than MoE. Single expert out of N is better than model of the same size (i.e. same as expert). 2 experts are better than one. That was on small LLM, not sure if it scales.
[ Parent | on HN ]

There are no comments on this comment yet.