News for hackers. Stuff that matters.
Re:RWKV Language Model by nullc on Thursday January 02, 2025 @12:52AM (#42572039)
Anyone ever look at doing a MoE like composition with RWKV and a transformer?
Re:RWKV Language Model by pico_creator on Thursday January 02, 2025 @02:19AM (#42572451)
Not an MoE, but we have already done hybrid models. And found it to be highly performant (as per the training budget)

https://arxiv.org/abs/2407.12077