News for hackers. Stuff that matters.
Re:The Secret Sauce behind 100K context window in LLMs: all ... by a2128 on Sunday June 18, 2023 @03:48AM (#36378156)
You can cache transformers though? Although the cache grows in size as more input tokens are added, while RWKV has to keep it in a single hidden state that's always the same size, but it still speeds up inference.

The huggingface transformers library exposes this as past_key_values, here's an example on GPT-2: https://huggingface.co/docs/transformers/model_doc/gpt2#tran...

There are no comments on this comment yet.