News for hackers. Stuff that matters.
Re:The Coming of Local LLMs by sebzim4500 on Tuesday April 11, 2023 @12:39PM (#35527597)
Probably once context lengths get really long it might be better to use ANN rather than exact attention at inference time. I would imagine this would only pay off with ~100,000 token contexts, and even then it would only work if only a few tokens meaningfully contribute to each attention head.

There are no comments on this comment yet.