News for hackers. Stuff that matters.
Re:Researchers run high-performing LLM on the energy needed ... by Aurornis on Tuesday June 25, 2024 @08:58AM (#40788024)
The press release is devoid of useful information, unsurprisingly. You can run an LLM under almost any energy envelope if you’re willing to wait long enough for the result. Total energy consumed and the time difference are the more important metrics.

The actual paper is here: https://arxiv.org/abs/2406.02528

The key part from the summary:

> To properly quantify the efficiency of our architecture, we build a custom hardware solution on an FPGA which exploits lightweight operations beyond what GPUs are capable of. We processed billion-parameter scale models at 13W beyond human readable throughput, moving LLMs closer to brain-like efficiency.

There is a lot of unnecessary obfuscation of the numbers going on in the abstract as well, which is unfortunate. Instead of quoting the numbers they call it “billion-parameter scale” and “beyond human readable throughout”.

Re:Researchers run high-performing LLM on the energy needed ... by FpUser on Tuesday June 25, 2024 @09:01AM (#40788047)
They did say that the answer is being produced faster than the human can read.
Re:Researchers run high-performing LLM on the energy needed ... by boto3 on Tuesday June 25, 2024 @09:59AM (#40788730)
yes, human average reading speed is ~250 words per minute, but to be fair, it's not a widely known stat.
Re:Researchers run high-performing LLM on the energy needed ... by onlyrealcuzzo on Tuesday June 25, 2024 @10:19AM (#40788981)
It's almost as if our brain is optimized for taking in more signals than just words on a page.
Re:Researchers run high-performing LLM on the energy needed ... by mensetmanusman on Tuesday June 25, 2024 @10:17AM (#40788961)
Speaking is 150 wpm
Re:Researchers run high-performing LLM on the energy needed ... by kevmo on Tuesday June 25, 2024 @10:02AM (#40788769)
250 seems high? I want to believe, though.
Re:Researchers run high-performing LLM on the energy needed ... by BenjiWiebe on Tuesday June 25, 2024 @10:04AM (#40788804)
That's 4 words per second.
Re:Researchers run high-performing LLM on the energy needed ... by pbnjay on Tuesday June 25, 2024 @09:58AM (#40788712)
Yeah... They are using a single-core 13W measurement to project out. For a 64x parallelization - no mention of any overhead due to parallelization or power needs of the supporting hardware. This is a key quote for me (page 12 of the PDF):

> The 1.3B parameter model, where L = 24 and d = 2048, has a projected runtime of 42ms, and a throughput of 23.8 tokens per second.

e.g. 64 x 13.67W = 874 Watts to run a 1.3B model at 23.8 t/s... I'm pretty sure my phone can do way better than that! Even half that power given their assertions in the table are still overpowered for such a small model.

Re:Researchers run high-performing LLM on the energy needed ... by shivaluminaire on Monday July 01, 2024 @02:48PM (#40849036)
When you multiply by 64 you also get 64 times more tokens per second!! Your math is wrong.
Re:Researchers run high-performing LLM on the energy needed ... by shivaluminaire on Monday July 01, 2024 @02:49PM (#40849050)
When you multiply by 64 you also get 64 times more tokens per second!! Your math is wrong.
Re:Researchers run high-performing LLM on the energy needed ... by pbnjay on Wednesday July 10, 2024 @11:44AM (#40928200)
That's their math, the 23.8t/s is already the 64x but they didn't 64x the other stats.