Re:Lossless Acceleration of LLM via Adaptive N-Gram Parallel...
by outofpaper on Sunday April 21, 2024 @03:44PM
(#40108607)
I think a number of us are using the method and these folks decided to do a paper and release their findings.
[ Parent
| on HN ]
Re:Lossless Acceleration of LLM via Adaptive N-Gram Parallel...
by jimmySixDOF on Monday April 22, 2024 @03:28AM
(#40112110)
Lots of speedups already on Llama3 like Groq (hardware arch) and Modal [1] getting 200 t/s
From the below Modal link they use: >continuous batching, so multiple generations can take place at the same time on a single container
>PagedAttention, which applies memory paging to the attention mechanism’s key-value cache, increasing throughput
[1]https://modal.com/docs/examples/text_generation_inference