Re:The Secret Sauce behind 100K context window in LLMs: all ...
by a2128 on Sunday June 18, 2023 @03:48AM
(#36378156)
You can cache transformers though? Although the cache grows in size as more input tokens are added, while RWKV has to keep it in a single hidden state that's always the same size, but it still speeds up inference.
The huggingface transformers library exposes this as past_key_values, here's an example on GPT-2: https://huggingface.co/docs/transformers/model_doc/gpt2#tran...