haeggee
Re:Implementation of mixture of experts language model in a single fil...
on Saturday March 23, 2024 @06:49AM (#39798926)
very cool work! we did something similar in the context of the Swiss AI initiative (https://www.swiss-ai.org/) here: https://github.com/swiss-ai/MoE. The implementation is as simple and fast as nanoGPT and works with our modular llm-baselines codebase (https://github.com/epfml/llm-baselines) for experimenting with transformers and different datasets :)