News for hackers. Stuff that matters.

A comment on Mixtures of Experts.

Re:Mixtures of Experts by ilaksh on Sunday October 08, 2023 @04:20PM (#37814065)
Is there a popular and performant open source model that uses mixture of experts?
[ Parent | on HN ]
Re:Mixtures of Experts by javidlakha on Monday October 09, 2023 @03:35PM (#37824416)
Google have released the models and code for the Switch Transformer from Fedus et al. (2021) under the Apache 2.0 licence. [0]

There's also OpenMoE - an open-source effort to train a mixture of experts model. Currently they've released a model with 8 billion parameters. [1]

[0] https://github.com/google-research/t5x/blob/main/docs/models...

[1] https://github.com/XueFuzhao/OpenMoE

[ Parent | on HN ]