News for hackers. Stuff that matters.
Re:New models and developer products announced at OpenAI DevDay by coder543 on Monday November 06, 2023 @03:13PM (#38168282)
I’m not the one who decided to that is the best comparison. The person I replied to is comparing Mistral to what was launched. I’m just going along with their comparison and showing how I have been unsatisfied with it, but these open models can certainly be fun to play with. Even 70B parameter models don’t do much better than Mistral in a lot of cases. I have no effective way of running Falcon-180B, unfortunately, so I can’t say how that does.

Regardless, where did you find 1.8T for GPT-4 Turbo? The Turbo model is the one with the 128K context size, and the Turbo models tend to have a much lower parameter count from what people can tell. Nobody outside of OpenAI even knows how many parameters regular GPT-4 has. 1.8T is one of several guesses I have seen people make, but the guesses vary significantly.

I’m also not convinced that parameter counts are everything, as your comment clearly implies, or that chinchilla scaling is fully understood. More research seems required to find the right balance: https://espadrine.github.io/blog/posts/chinchilla-s-death.ht...

Re:New models and developer products announced at OpenAI DevDay by danielmarkbruce on Monday November 06, 2023 @03:38PM (#38168643)
It's an order of magnitude comparison.

Let's just agree it's 100x-300x more parameters, and let's assume the open ai folks are pretty smart and have a sense for the optimal number of tokens to train on.

Re:New models and developer products announced at OpenAI DevDay by razodactyl on Monday November 06, 2023 @03:49PM (#38168793)
This definitely. Andrej Karpathy himself mentions tuned weight initialisation in one of his lectures. The TinyGPT code he wrote goes through it.

Additionally explanations for the raw mathematics of log likelihoods and their loss ballparks.

Interesting low-level stuff. These researchers are the best of the best working for the company that can afford them working on the best models available.

Re:New models and developer products announced at OpenAI DevDay by razodactyl on Monday November 06, 2023 @03:45PM (#38168721)
Nah, it's training quality and context saturation.

Grab an 8K context model, tweak some internals and try to pass 32K context into it - it's still an 8K model and will go glitchy beyond 8K unless it's trained at higher context lengths.

Anthropic for example talk about the model's ability to spot words in the entire Great Gatsby novel loaded into context. It's a hint to how the model is trained.

Parameter counts are a unified metric, what seems to be important is embedding dimensionality to transfer information through the layers - and the layers themselves to both store and process the nuance of information.