Re:Implementation of mixture of experts language model in a ...
by haeggee on Saturday March 23, 2024 @06:49AM
(#39798926)
very cool work! we did something similar in the context of the Swiss AI initiative (https://www.swiss-ai.org/) here: https://github.com/swiss-ai/MoE. The implementation is as simple and fast as nanoGPT and works with our modular llm-baselines codebase (https://github.com/epfml/llm-baselines) for experimenting with transformers and different datasets :)
[ Parent
| on HN ]
Re:Implementation of mixture of experts language model in a ...
by avisoori1x on Saturday March 23, 2024 @02:13PM
(#39801936)
This is awesome! Thanks for sharing. I'll definitely check this out.