Re:Serving 70B-Scale LLMs Efficiently on Low-Resource Edge D...
by _ache_ on Thursday October 03, 2024 @12:19PM
(#41732127)
That is s/token and not token/s. The cost is high.
The actual goal of the article is to highlight that we can optimise the overall speed by decreasing link latency. Yeah link latency, because it's not 1 machine but several low devices that are used together to serve the 70B LLM.
The actual goal of the article is to highlight that we can optimise the overall speed by decreasing link latency. Yeah link latency, because it's not 1 machine but several low devices that are used together to serve the 70B LLM.