Re:RWKV Language Model
by nullc on Thursday January 02, 2025 @12:52AM
(#42572039)
Anyone ever look at doing a MoE like composition with RWKV and a transformer?
[ Parent
| on HN ]
Re:RWKV Language Model
by pico_creator on Thursday January 02, 2025 @02:19AM
(#42572451)
Not an MoE, but we have already done hybrid models. And found it to be highly performant (as per the training budget)