RWKV Language Model

Re:RWKV Language Model by intalentive on Thursday January 02, 2025 @12:55AM (#42572060)

Idea for killer app for recurrent models: low latency, low memory LLM / TTS coupling. Start decoding / generating speech as soon as new tokens are generated. When the LLM is cranking out token t, the TTS is already working on token t-1. It doesn’t have to wait. Then when the LLM is finished, the TTS is nearly finished too. The two models being colocated you just saved another network call as well.

Recurrent models with constant hidden state are naturally suited to streaming data, potentially opening the door to unexplored new use cases.

[ Parent | on HN ]

Re:RWKV Language Model by computerex on Thursday January 02, 2025 @01:05AM (#42572101)

New multimodal models take raw speech input and provide raw speech output, no tts in the middle.

[ Parent | on HN ]

Re:RWKV Language Model by Closi on Thursday January 02, 2025 @04:02AM (#42572864)

Seems like the future - so much meaning and context is lost otherwise.

[ Parent | on HN ]

Re:RWKV Language Model by benob on Thursday January 02, 2025 @02:39AM (#42572548)

A relatively detailed description of such systems: https://arxiv.org/abs/2410.00037

[ Parent | on HN ]

Re:RWKV Language Model by intalentive on Thursday January 02, 2025 @04:51AM (#42573119)

Very cool. Logical next step. Would be interested to know what the dataset looks like.

[ Parent | on HN ]

Re:RWKV Language Model by moffkalast on Thursday January 02, 2025 @05:51AM (#42573401)

Youtube. Youtube is the dataset.

[ Parent | on HN ]

Re:RWKV Language Model by pico_creator on Thursday January 02, 2025 @02:22AM (#42572466)

This is actually the hypothesis for cartesia (state space team), and hence their deep focus on voice model specifically. Taking full advantage of recurrent models constant time compute, for low latencies.

RWKV team's focus is still however is first in the multi-lingual text space, then multi-modal space in the future.

[ Parent | on HN ]

Re:RWKV Language Model by swyx on Thursday January 02, 2025 @02:40AM (#42572550)

Karan from Cartesia explains SSMs+voice really well: https://www.youtube.com/watch?v=U9DPRZ0lSIQ

its one of those retrospectively obvious/genius insights that i wish i understood when i first met him

[ Parent | on HN ]

Re:RWKV Language Model by cootsnuck on Thursday January 02, 2025 @01:33AM (#42572246)

This can currently already be done using a streaming capable LLM with a streaming input/output TTS model.

[ Parent | on HN ]

Re:RWKV Language Model by lostmsu on Thursday January 02, 2025 @04:48AM (#42573103)

Any LLM is "streaming capable".

[ Parent | on HN ]

Re:RWKV Language Model by Xmd5a on Thursday January 02, 2025 @08:45AM (#42574340)

https://github.com/mit-han-lab/streaming-llm

On a side node, and that's what led me to the link above, I wonder if it would be possible to chain N streaming LLMs in an agent workflow and get a final output stream almost instantaneously without waiting for N-1 LLM to complete their reply.

[ Parent | on HN ]

Re:RWKV Language Model by yshui on Thursday January 02, 2025 @08:57AM (#42574424)

Any autoregressive model can do what you are describing. transformers are generating one token at a time too, not all at once.

[ Parent | on HN ]

Re:RWKV Language Model by whimsicalism on Thursday January 02, 2025 @10:10AM (#42575065)

yes but transformers are much slower than state space models

[ Parent | on HN ]

Re:RWKV Language Model by intalentive on Thursday January 02, 2025 @02:59PM (#42578156)

True but memory requirements grow with sequence length. For recurrent models the memory requirement is constant. This is why I qualified with "low memory".

[ Parent | on HN ]