News for hackers. Stuff that matters.
Re:Meta Llama 3 by bearjaws on Thursday April 18, 2024 @02:12PM (#40079066)
My guess is around 256GiB but it depends on what level of quantization you are okay with. At full 16bit it will be massive, near 512GiB.

I figure we will see some Q4's that can probably fit on 4 4090s with CPU offloading.

Re:Meta Llama 3 by sp332 on Thursday April 18, 2024 @02:21PM (#40079161)
With 400 billion parameters and 8 bits per parameter, wouldn't it be ~400 GB? Plus context size which could be quite large.
Re:Meta Llama 3 by yalok on Thursday April 18, 2024 @03:03PM (#40079619)
he said "Q4" - meaning 4-bit weights.
Re:Meta Llama 3 by sp332 on Thursday April 18, 2024 @03:18PM (#40079774)
Ok but at 16-bit it would be 800GB+, right? Not 512.
Re:Meta Llama 3 by reactordev on Thursday April 18, 2024 @06:12PM (#40081310)
Divide not multiply. If a size is estimated in 8-bit, reducing to 4-bit halves the size (and entropy of each value). Difference between INT_MAX and SHORT_MAX (assuming you have such defs).

I could be wrong too but that’s my understanding. Like float vs half-float.