Re:Researchers run high-performing LLM on the energy needed ...
by FpUser on Tuesday June 25, 2024 @09:01AM
(#40788047)
They did say that the answer is being produced faster than the human can read.
Re:Researchers run high-performing LLM on the energy needed ...
by boto3 on Tuesday June 25, 2024 @09:59AM
(#40788730)
yes, human average reading speed is ~250 words per minute, but to be fair, it's not a widely known stat.
Re:Researchers run high-performing LLM on the energy needed ...
by onlyrealcuzzo on Tuesday June 25, 2024 @10:19AM
(#40788981)
It's almost as if our brain is optimized for taking in more signals than just words on a page.
Re:Researchers run high-performing LLM on the energy needed ...
by mensetmanusman on Tuesday June 25, 2024 @10:17AM
(#40788961)
Speaking is 150 wpm
Re:Researchers run high-performing LLM on the energy needed ...
by kevmo on Tuesday June 25, 2024 @10:02AM
(#40788769)
250 seems high? I want to believe, though.
Re:Researchers run high-performing LLM on the energy needed ...
by BenjiWiebe on Tuesday June 25, 2024 @10:04AM
(#40788804)
That's 4 words per second.
Re:Researchers run high-performing LLM on the energy needed ...
by pbnjay on Tuesday June 25, 2024 @09:58AM
(#40788712)
Yeah... They are using a single-core 13W measurement to project out. For a 64x parallelization - no mention of any overhead due to parallelization or power needs of the supporting hardware. This is a key quote for me (page 12 of the PDF):
> The 1.3B parameter model, where L = 24 and d = 2048, has a projected runtime of 42ms, and a throughput of 23.8 tokens per second.
e.g. 64 x 13.67W = 874 Watts to run a 1.3B model at 23.8 t/s... I'm pretty sure my phone can do way better than that! Even half that power given their assertions in the table are still overpowered for such a small model.
Re:Researchers run high-performing LLM on the energy needed ...
by shivaluminaire on Monday July 01, 2024 @02:48PM
(#40849036)
When you multiply by 64 you also get 64 times more tokens per second!! Your math is wrong.
Re:Researchers run high-performing LLM on the energy needed ...
by shivaluminaire on Monday July 01, 2024 @02:49PM
(#40849050)
When you multiply by 64 you also get 64 times more tokens per second!! Your math is wrong.
The actual paper is here: https://arxiv.org/abs/2406.02528
The key part from the summary:
> To properly quantify the efficiency of our architecture, we build a custom hardware solution on an FPGA which exploits lightweight operations beyond what GPUs are capable of. We processed billion-parameter scale models at 13W beyond human readable throughput, moving LLMs closer to brain-like efficiency.
There is a lot of unnecessary obfuscation of the numbers going on in the abstract as well, which is unfortunate. Instead of quoting the numbers they call it “billion-parameter scale” and “beyond human readable throughout”.