News for hackers. Stuff that matters.

A comment on Meta Llama 3. Parent comment

Re:Meta Llama 3 by tanelpoder on Friday April 19, 2024 @03:36AM (#40084281)
I just tested llama3:70b with ollama on my old AMD ThreadRipper Pro 3965WX workstation (16-core Zen4 with 8 DDR4 mem channels), with a single RTX 4090.

Got 3.5-4 tokens/s, GPU compute was <20% busy (~90W) and the 16 CPU cores / 32 threads were about 50% busy.

[ Parent | on HN ]
Re:Meta Llama 3 by llm_trw on Friday April 19, 2024 @08:40AM (#40086088)
Jesus that's the old one?
[ Parent | on HN ]
Re:Meta Llama 3 by reasonabl_human on Friday April 19, 2024 @11:44AM (#40088238)
And that’s not quantized at all, correct?

If so, then the parent comment’s sentiment holds true…. Exciting stuff.

[ Parent | on HN ]