News for hackers. Stuff that matters.
Re:RWKV Language Model by pico_creator on Thursday January 02, 2025 @02:19AM (#42572451)
Not an MoE, but we have already done hybrid models. And found it to be highly performant (as per the training budget)

https://arxiv.org/abs/2407.12077

There are no comments on this comment yet.