News for hackers. Stuff that matters.
Re:Let's Think Dot by Dot: Hidden Computation in Transformer... by diziet on Sunday April 28, 2024 @06:22AM (#40187463)
I understand what you are trying to say, but correct me if I am wrong: Each forward pass is stateless. The KV cache you describe is a clever method to make compute scale closer to linear complexity for attention. I am trying to build some conceptual understanding of how performance on 3SUM problem improves on this fine-tuned example but not on the fine-tuned no shot.
Re:Let's Think Dot by Dot: Hidden Computation in Transformer... by edmara on Sunday April 28, 2024 @06:27PM (#40192553)
In the case of 3sum, because the LLM has been fine tuned to use 'blank' token key values as a register to represent the sums of specific integer triplets.