News for hackers. Stuff that matters.
Re:Let's Think Dot by Dot: Hidden Computation in Transformer... by mmoskal on Saturday April 27, 2024 @10:46PM (#40185585)
Every token generates a KV cache entry based on this token and all previous KV cache entries. This happens in every layer. KV cache is 100k-1MB per token so quite a bit.

Edit: also you can forward generate 5 or 10 dots in batch without much overhead compared to a single dot since the main cost is pulling KV cache from VRAM so you have free tensor units.

Re:Let's Think Dot by Dot: Hidden Computation in Transformer... by diziet on Sunday April 28, 2024 @06:22AM (#40187463)
I understand what you are trying to say, but correct me if I am wrong: Each forward pass is stateless. The KV cache you describe is a clever method to make compute scale closer to linear complexity for attention. I am trying to build some conceptual understanding of how performance on 3SUM problem improves on this fine-tuned example but not on the fine-tuned no shot.
Re:Let's Think Dot by Dot: Hidden Computation in Transformer... by edmara on Sunday April 28, 2024 @06:27PM (#40192553)
In the case of 3sum, because the LLM has been fine tuned to use 'blank' token key values as a register to represent the sums of specific integer triplets.