News for hackers. Stuff that matters.
Re:New models and developer products announced at OpenAI DevDay by visarga on Monday November 06, 2023 @01:55PM (#38167031)
Mistral + 2 weeks of work from the community. Not as good, but private and free. It will trail OpenAI by 6-12 months in capabilities.
Re:New models and developer products announced at OpenAI DevDay by coder543 on Monday November 06, 2023 @02:17PM (#38167401)
OpenAI offering 128k context is very appealing, however.

I tried some Mistral variants with larger context windows, and had very poor results… the model would often offer either an empty completion or a nonsensical completion, even though the content fit comfortably within the context window, and I was placing a direct question either at the beginning or end, and either with or without an explanation of the task and the content. Large contexts just felt broken. There are so many ways that we are more than “two weeks” from the open source solutions matching what OpenAI offers.

And that’s to say nothing of how far behind these smaller models are in terms of accuracy or instruction following.

For now, 6-12 months behind also isn’t good enough. In the uncertain case that this stays true, then a year from now the open models could be perfectly adequate for many use cases… but it’s very hard to predict the progression of these technologies.

Re:New models and developer products announced at OpenAI DevDay by pclmulqdq on Monday November 06, 2023 @03:06PM (#38168166)
Comparing a 7B parameter model to a 1.8T parameter model is kind of silly. Of course it's behind on accuracy, but it also takes 1% of the resources.
Re:New models and developer products announced at OpenAI DevDay by coder543 on Monday November 06, 2023 @03:13PM (#38168282)
I’m not the one who decided to that is the best comparison. The person I replied to is comparing Mistral to what was launched. I’m just going along with their comparison and showing how I have been unsatisfied with it, but these open models can certainly be fun to play with. Even 70B parameter models don’t do much better than Mistral in a lot of cases. I have no effective way of running Falcon-180B, unfortunately, so I can’t say how that does.

Regardless, where did you find 1.8T for GPT-4 Turbo? The Turbo model is the one with the 128K context size, and the Turbo models tend to have a much lower parameter count from what people can tell. Nobody outside of OpenAI even knows how many parameters regular GPT-4 has. 1.8T is one of several guesses I have seen people make, but the guesses vary significantly.

I’m also not convinced that parameter counts are everything, as your comment clearly implies, or that chinchilla scaling is fully understood. More research seems required to find the right balance: https://espadrine.github.io/blog/posts/chinchilla-s-death.ht...

Re:New models and developer products announced at OpenAI DevDay by danielmarkbruce on Monday November 06, 2023 @03:38PM (#38168643)
It's an order of magnitude comparison.

Let's just agree it's 100x-300x more parameters, and let's assume the open ai folks are pretty smart and have a sense for the optimal number of tokens to train on.

Re:New models and developer products announced at OpenAI DevDay by razodactyl on Monday November 06, 2023 @03:49PM (#38168793)
This definitely. Andrej Karpathy himself mentions tuned weight initialisation in one of his lectures. The TinyGPT code he wrote goes through it.

Additionally explanations for the raw mathematics of log likelihoods and their loss ballparks.

Interesting low-level stuff. These researchers are the best of the best working for the company that can afford them working on the best models available.

Re:New models and developer products announced at OpenAI DevDay by razodactyl on Monday November 06, 2023 @03:45PM (#38168721)
Nah, it's training quality and context saturation.

Grab an 8K context model, tweak some internals and try to pass 32K context into it - it's still an 8K model and will go glitchy beyond 8K unless it's trained at higher context lengths.

Anthropic for example talk about the model's ability to spot words in the entire Great Gatsby novel loaded into context. It's a hint to how the model is trained.

Parameter counts are a unified metric, what seems to be important is embedding dimensionality to transfer information through the layers - and the layers themselves to both store and process the nuance of information.

Re:New models and developer products announced at OpenAI DevDay by visarga on Monday November 06, 2023 @05:07PM (#38169755)
Usually the 7B model is fine-tuned with "enriched" data, "textbook quality" generations from the 1.8T model. Riding on its coat tails.
Re:New models and developer products announced at OpenAI DevDay by tannhaeuser on Monday November 06, 2023 @04:05PM (#38168996)
That's my take-away from limited attempts to get Code Llama2 Instruct to implement a moderately complex spec as well, using special INST and SYS tokens even or just pasting some spec text along in a 12k context when Code Llama2 supposedly can honor up to 100k tokens. And I don't even know how to combine code infilling with an elaborate spec text exceeding the volume of what normally goes into code comments. Is ChatGPT 4 really any better?
Re:New models and developer products announced at OpenAI DevDay by fancy_pantser on Monday November 06, 2023 @05:41PM (#38170170)
Open researchers are trying to shrink and speed up 138K models e.g. YaRN https://github.com/jquesnelle/yarn

It's very compelling and opens up a lot of use cases, so I've been keeping an eye out for advancements. However, inferencing on 4xA100s would be the target today for YaRN and 128K to get a reasonable token rate on their version of Mistral.