Llama 3 400B Base / Instruct
MMLU 84.8 86.1
GPQA - 48.0
MATH - 57.8
HumanEval - 84.1
DROP 83.5 -Although 400B will be pretty much out of reach for any PC to run locally, it will still be exciting to have a GPT-4 level model in the open for research so people can try quantizing, pruning, distilling, and other ways of making it more practical to run. And I'm sure startups will build on it as well.
Still wouldn't be super performant AFA token gen, ~4-6 per second, but certainly runnable.
Of course by the time that lands in 6-12 months we'll probably have a 70-100G model that is similarly performant.
Llama 3 GPT 4(Published)
BBH 85.3 83.1
MMLU 86.1 86.4
DROP 83.5 80.9
GSM8K 94.1 92.0
MATH 57.8 52.9
HumEv 84.1 74.4
Although it should be noted that the API numbers were generally better than published numbers for GPT4.That said, I am very curious what OpenAI has in their labs... Are they actually barely ahead? Or do they have something much better that is not yet public? Perhaps they were waiting for Llama 3 to show it? Exciting times ahead either way!
While all the others are catching up and in some cases being slightly better, I wouldn't be surprised to see a rather large leap back into the lead from OpenAI pretty soon and then a scrabble for some time for others to get close again. We will really see who has the momentum soon, when we see OpenAI's next full release.
I figure we will see some Q4's that can probably fit on 4 4090s with CPU offloading.
I could be wrong too but that’s my understanding. Like float vs half-float.
Llama 3 GPT-4 GPT-4-Turbo* (Apr 2024)
MMLU 86.1 86.4 86.7
DROP 83.5 80.9 86.0
MATH 57.8 52.9 73.4
HumEv 84.1 74.4 88.2
*using API prompt: https://github.com/openai/simple-evalsThere were times when I felt that too, but nowadays I predominantly use turbo. It's probably because turbo is faster and cheaper, but in lmsys turbo has 100 elo higher than original, so by and large people simply find turbo to be....better?
Nevertheless, I do wonder if not just in benchmarks but in how people use LLMs, intelligence is somewhat under utilised, or possibly offset by other qualities.
Since price influences my calculus, I can't say this for sure, but it seems being slightly smarter is not much of an edge, because it's still dumb by human standards. For most non-coding use the smart doesn't make much difference (like summarisation), I find that cheaper options like mistral-large do just as good as Opus.
In the last month I have used Command R+ more and more. Finally had some excuse to write some function calling stuff. I have also been highly impressed by Gemini Pro 1.5 finding technical answers from a dense 650 page pdf manual. I have enjoyed chatting with the WizardLM2 fine-tune for the past few days.
Somehow I haven't quite found a consistent use case for Opus.
HumanEval is meaningless regardless, those 164 problems have been overfit to the tea.
Hook this up to LLM arena we will get a better picture regarding how powerful they really are
Ahhh that takes me back!
In the keynote, Jensen uses 1.8T in an example and suggests that this is roughly the size of GPT-4 (if I remember correctly).
That might suggest that GPT-4 should be thought of as something like a 250B model. But there's also some selection for the remaining 1/8 of weights that are used by the chosen expert as being the "most useful" weights for that pass (as chosen/defined by the mixture routing), so now it feels like 250B is undercounting the parameter size, whereas 1.8T was overcounting it.
I think it's not really defined how to compare parameter counts with a MoE model.
Similarly, if GPT-4 is really 1.8T you would expect it to produce output of similar quality to a comparable 1.8T model without MoE architecture.
Are you sure about that? I'm pretty sure Miqu (the leaked Mistral 70b model) is generally thought to be smarter than Mixtral 8x7b.
From Databricks: "DBRX has 16 experts and chooses 4, while Mixtral and Grok-1 have 8 experts and choose 2. This provides 65x more possible combinations of experts and we found that this improves model quality. DBRX uses rotary position encodings (RoPE), gated linear units (GLU), and grouped query attention (GQA). It uses the GPT-4 tokenizer as provided in the tiktoken repository. We made these choices based on exhaustive evaluation and scaling experiments."
https://www.databricks.com/blog/introducing-dbrx-new-state-a...
Everyone seems to put each expert on a different GPU in training and inference, so that's how you get to 8 experts, or 7 if you want to put the router on its own GPU too.
You could also do multiples of 8. But from my limited understanding it seems like more experts don't perform better. The main advantage of MoE is the ability to split the model into parts that don't talk to each other, and run these parts in different GPUs or different machines.
1. After ChatGPT release, Twitter spam from influencers about chatGPT is one billion and GPT-4 is 1 trillion.
2. Semianalysis publishes a blog post claiming 1.8T sourced from insiders.
3. The way info diffusion works these days, everyone heard from someone else other than Semianalysis.
4. Up until about a month ago, you could confidently say "hey its just that one blog post" and work through it with people to trace their initial hearing of it back to the post.
5. nVidia press conference some time in the last month used the rumors as an example with "apparently" attached, and now people will tell you NVidia confirmed 1.8 trillion.
my $0.02: I'd bet my life GPT-4 isn't 1.8T, and I very much doubt its over 1 trillion. Like, lightning striking the same person 3 times in the same week.
In general, not someone you wanna be citing with lengthy platitudes, he's an influencer who speaks engineer, he's burned out of every community he's been in, acrimonously.
But it's pretty clear GPT4 Turbo is a smaller and heavily quantized model.
I just added Llama 3 70B to our coding copilot https://www.double.bot if anyone wants to try it for coding within their IDE
GPT-4 numbers from from https://github.com/openai/simple-evals gpt-4-turbo-2024-04-09 (chatgpt)