Re:The Coming of Local LLMs
by sebzim4500 on Tuesday April 11, 2023 @12:39PM
(#35527597)
Probably once context lengths get really long it might be better to use ANN rather than exact attention at inference time. I would imagine this would only pay off with ~100,000 token contexts, and even then it would only work if only a few tokens meaningfully contribute to each attention head.
[ Parent
| on HN ]
There are no comments on this comment yet.