Re:Implementation of mixture of experts language model in a ...
by zingelshuher on Friday March 22, 2024 @05:25PM
(#39795027)
It was inspired by Mixtral 8x7B, of course. I think the same approach, soft to hard MoE, can be used in other domains. Like video/image processing. Would be interesting to take it to extreme, like 4 experts out of 100.
[ Parent
| on HN ]
There are no comments on this comment yet.