RWKV Language Model

Re:RWKV Language Model by sushidev on Thursday January 02, 2025 @02:10AM (#42572417)

Interesting. Very cryptic for simple user like me. I wonder if it’s useful today and for what purposes

[ Parent | on HN ]

Re:RWKV Language Model by pico_creator on Thursday January 02, 2025 @02:25AM (#42572479)

Currently the strongest RWKV model is 32B in size: https://substack.recursal.ai/p/q-rwkv-6-32b-instruct-preview

This is a full drop in replacement for any transformer model use cases on model sizes 32B and under, as it has equal performance to existing open 32B models in most benchmarks

We are in works on a 70B, which will be a full drop in replacement for most text use cases

[ Parent | on HN ]

Re:RWKV Language Model by swyx on Thursday January 02, 2025 @02:39AM (#42572544)

how about finetuning your 32B to be R1QWQKV?

[ Parent | on HN ]

Re:RWKV Language Model by pico_creator on Thursday January 02, 2025 @07:30AM (#42573857)

There is a current lack of "O1 style" reasoning dataset in open source space. QWQ did not release their dataset. So that would take some time for the community to prepare.

It's definitely something we are tracking to do as well =)

[ Parent | on HN ]

Re:RWKV Language Model by lostmsu on Thursday January 02, 2025 @04:52AM (#42573126)

Why aren't you on lmarena (former chatbot arena) leaderboard?

[ Parent | on HN ]

Re:RWKV Language Model by pico_creator on Thursday January 02, 2025 @07:28AM (#42573850)

kinda on a todo list, the model is open source on HF for anyone who is willing to make it work with lmarena

[ Parent | on HN ]