News for hackers. Stuff that matters.
Re:Meta Llama 3 by sp332 on Thursday April 18, 2024 @12:52PM (#40078229)
The bottom of https://ai.meta.com/blog/meta-llama-3/ has in-progress results for the 400B model as well. Looks like it's not quite there yet.

  Llama 3 400B Base / Instruct
  MMLU         84.8   86.1
  GPQA          -     48.0
  MATH          -     57.8
  HumanEval     -     84.1
  DROP         83.5    -
Re:Meta Llama 3 by modeless on Thursday April 18, 2024 @02:23PM (#40079177)
Not quite there yet, but very close and not done training! It's quite plausible that this model could be state of the art over GPT-4 in some domains when it finishes training, unless GPT-5 comes out first.

Although 400B will be pretty much out of reach for any PC to run locally, it will still be exciting to have a GPT-4 level model in the open for research so people can try quantizing, pruning, distilling, and other ways of making it more practical to run. And I'm sure startups will build on it as well.

Re:Meta Llama 3 by int_19h on Thursday April 18, 2024 @07:45PM (#40081923)
The real question will be, how much you can quantize that while still retaining sanity. 400b at 2-bit would be possible to run on a Mac Studio - probably at multiple seconds per token, but sometimes that's "fast enough".
Re:Meta Llama 3 by modeless on Thursday April 18, 2024 @07:48PM (#40081941)
Yes. I expect an explosion of research and experimentation in model compression. The good news is I think there are tons of avenues that have barely been explored at all. We are at the very beginning of understanding this stuff, and my bet is that in a few years we'll be able to compress these models 10x or more.
Re:Meta Llama 3 by brandall10 on Friday April 19, 2024 @11:32AM (#40088090)
There are rumors about an upcoming M3 or M4 Extreme chip... which would certainly have enough RAM, and probably a 1600-2000 GB/s bandwidth.

Still wouldn't be super performant AFA token gen, ~4-6 per second, but certainly runnable.

Of course by the time that lands in 6-12 months we'll probably have a 70-100G model that is similarly performant.

Re:Meta Llama 3 by YetAnotherNick on Thursday April 18, 2024 @12:57PM (#40078294)
For the still training 400B:

          Llama 3 GPT 4(Published)
    BBH   85.3    83.1
    MMLU  86.1    86.4
    DROP  83.5    80.9
    GSM8K 94.1    92.0    
    MATH  57.8    52.9
    HumEv 84.1    74.4
Although it should be noted that the API numbers were generally better than published numbers for GPT4.

[1]: https://deepmind.google/technologies/gemini/

Re:Meta Llama 3 by oliwary on Thursday April 18, 2024 @01:23PM (#40078561)
Wild! So if this indeed holds up, it looks like OpenAI were about a year ahead when GPT-4 was released, compared to the open source world. However, given the timespan between matching GPT-3.5 (Mixtral perhaps?) and matching GPT-4 has just been a few weeks, I am wondering if the open source models have more momentum.

That said, I am very curious what OpenAI has in their labs... Are they actually barely ahead? Or do they have something much better that is not yet public? Perhaps they were waiting for Llama 3 to show it? Exciting times ahead either way!

Re:Meta Llama 3 by ChildOfChaos on Thursday April 18, 2024 @06:50PM (#40081573)
You've also got to consider that we don't really know where OpenAI are though, what they have released in the past year have been tweaks to GPT4, while I am sure the real work is going into GPT5 or whatever it gets called.

While all the others are catching up and in some cases being slightly better, I wouldn't be surprised to see a rather large leap back into the lead from OpenAI pretty soon and then a scrabble for some time for others to get close again. We will really see who has the momentum soon, when we see OpenAI's next full release.

Re:Meta Llama 3 by mdeeks on Thursday April 18, 2024 @02:46PM (#40079426)
Which specific GPT-4 model is this? gpt-4-0613? gpt-4-0125-preview?
Re:Meta Llama 3 by YetAnotherNick on Thursday April 18, 2024 @02:58PM (#40079557)
This is mostly from technical report from OpenAI[1]. API performs better as I said in my previous comment. API models(0613/0125 etc.) also uses user data for training which could leak the benchmark data.

[1]: https://arxiv.org/pdf/2303.08774.pdf

Re:Meta Llama 3 by pama on Thursday April 18, 2024 @04:45PM (#40080522)
IIRC this model had finished pretraining in the summer of 2022.
Re:Meta Llama 3 by tmikaeld on Thursday April 18, 2024 @01:33PM (#40078655)
Hm, how much VRAM would this take to run?
Re:Meta Llama 3 by bearjaws on Thursday April 18, 2024 @02:12PM (#40079066)
My guess is around 256GiB but it depends on what level of quantization you are okay with. At full 16bit it will be massive, near 512GiB.

I figure we will see some Q4's that can probably fit on 4 4090s with CPU offloading.

Re:Meta Llama 3 by sp332 on Thursday April 18, 2024 @02:21PM (#40079161)
With 400 billion parameters and 8 bits per parameter, wouldn't it be ~400 GB? Plus context size which could be quite large.
Re:Meta Llama 3 by yalok on Thursday April 18, 2024 @03:03PM (#40079619)
he said "Q4" - meaning 4-bit weights.
Re:Meta Llama 3 by sp332 on Thursday April 18, 2024 @03:18PM (#40079774)
Ok but at 16-bit it would be 800GB+, right? Not 512.
Re:Meta Llama 3 by reactordev on Thursday April 18, 2024 @06:12PM (#40081310)
Divide not multiply. If a size is estimated in 8-bit, reducing to 4-bit halves the size (and entropy of each value). Difference between INT_MAX and SHORT_MAX (assuming you have such defs).

I could be wrong too but that’s my understanding. Like float vs half-float.

Re:Meta Llama 3 by asadm on Thursday April 18, 2024 @01:39PM (#40078712)
yes
Re:Meta Llama 3 by mrtranscendence on Thursday April 18, 2024 @02:09PM (#40079025)
Back of the envelope, maybe 0.75TB? More than you have, probably ...
Re:Meta Llama 3 by kyboren on Thursday April 18, 2024 @08:00PM (#40082002)
"More than you can afford, pal--NVidia."
Re:Meta Llama 3 by tedsanders on Thursday April 18, 2024 @05:47PM (#40081089)
Those numbers are for the original GPT-4 (Mar 2023). Current GPT-4-Turbo (Apr 2024) is better:

          Llama 3 GPT-4   GPT-4-Turbo* (Apr 2024)
    MMLU  86.1    86.4    86.7
    DROP  83.5    80.9    86.0
    MATH  57.8    52.9    73.4
    HumEv 84.1    74.4    88.2
*using API prompt: https://github.com/openai/simple-evals
Re:Meta Llama 3 by natrys on Thursday April 18, 2024 @06:32PM (#40081443)
I find it somewhat interesting that there is a common perception about GPT-4 at release being actually smart, but that it got gradually nerfed for speed with turbo, which is better tuned but doesn't exhibit intelligence like the original.

There were times when I felt that too, but nowadays I predominantly use turbo. It's probably because turbo is faster and cheaper, but in lmsys turbo has 100 elo higher than original, so by and large people simply find turbo to be....better?

Nevertheless, I do wonder if not just in benchmarks but in how people use LLMs, intelligence is somewhat under utilised, or possibly offset by other qualities.

Re:Meta Llama 3 by thelittleone on Thursday April 18, 2024 @06:44PM (#40081538)
Have you tried Claude 3 Opus? I've been using that predominantly since release and find it's "smarts" as or better than my experience with GPT-4 (pre turbo).
Re:Meta Llama 3 by natrys on Thursday April 18, 2024 @07:12PM (#40081719)
I did. It definitely exudes more all around personality. Unfortunately in my private test suite (mostly about coding), it did somewhat worse than turbo or phind 70b.

Since price influences my calculus, I can't say this for sure, but it seems being slightly smarter is not much of an edge, because it's still dumb by human standards. For most non-coding use the smart doesn't make much difference (like summarisation), I find that cheaper options like mistral-large do just as good as Opus.

In the last month I have used Command R+ more and more. Finally had some excuse to write some function calling stuff. I have also been highly impressed by Gemini Pro 1.5 finding technical answers from a dense 650 page pdf manual. I have enjoyed chatting with the WizardLM2 fine-tune for the past few days.

Somehow I haven't quite found a consistent use case for Opus.

Re:Meta Llama 3 by bugglebeetle on Thursday April 18, 2024 @06:50PM (#40081572)
Given the incremental increase between GPT-4 and its turbo variant, I would weight “vibes” more heavily than this improvement on MMLU. OpenAI isn’t exactly a very honest or transparent company and the metric is imperfect. As a longtime time user of ChatGPT, I observed it got markedly worse at coding after the turbo release, specifically in its refusal to complete code as specified.
Re:Meta Llama 3 by ljhskyso on Friday April 19, 2024 @11:15AM (#40087867)
i think it might just be the subjective feelings (GPT-4-turbo being dumber) - the joy is always stronger when you first taste it, and the joy decays as you get used to it and the bar raises ever since.
Re:Meta Llama 3 by jug on Thursday April 18, 2024 @05:48PM (#40081101)
This is tantalizingly close in multiple benchmarks though. Pretty sure this one will finally be the open GPT-4 match.