News for hackers. Stuff that matters.

A comment on Anthropic - Introducing 100K Token Context Windows, Around 75,000 Words. Parent comment

Re:Anthropic - Introducing 100K Token Context Windows, Aroun... by space_fountain on Thursday May 11, 2023 @01:38PM (#35905505)
Well each additional token generated requires rerunning the model right to find the next likely token given the previous one
[ Parent | on HN ]
Re:Anthropic - Introducing 100K Token Context Windows, Aroun... by newhouseb on Thursday May 11, 2023 @01:46PM (#35905607)
Naively, yes, but you can cache the bulk of that "rerunning" [1]. That said the (non-flash) attention costs go up with the length of the sequence so perhaps this is just a simpler way to approximate these costs.

[1] https://kipp.ly/blog/transformer-inference-arithmetic/

[ Parent | on HN ]