News for hackers. Stuff that matters.

A comment on Implementation of mixture of experts language model in a single file of PyTorch.

Re:Implementation of mixture of experts language model in a ... by zingelshuher on Friday March 22, 2024 @02:46PM (#39793553)
Similar MoE implementation was on GitHub for a while, since Jan 2024

https://github.com/zxaall/moegpt

[ Parent | on HN ]
Re:Implementation of mixture of experts language model in a ... by avisoori1x on Friday March 22, 2024 @02:50PM (#39793594)
Oh nice. What's new here would be noisy top-k routing and expert capacity. It also seems to use the nanoGPT base from Andrej Karpathy. Mine is from January as well. Here's the original blog: https://huggingface.co/blog/AviSoori1x/makemoe-from-scratch
[ Parent | on HN ]
Re:Implementation of mixture of experts language model in a ... by zingelshuher on Friday March 22, 2024 @05:25PM (#39795027)
It was inspired by Mixtral 8x7B, of course. I think the same approach, soft to hard MoE, can be used in other domains. Like video/image processing. Would be interesting to take it to extreme, like 4 experts out of 100.
[ Parent | on HN ]