News for hackers. Stuff that matters.
Re:Mixtral 8x22B by mdrzn on Wednesday April 17, 2024 @10:59AM (#40065620)
"64K tokens context window" I do wish they had managed to extend it to at least 128K to match the capabilities of GPT-4 Turbo

Maybe this limit will become a joke when looking back? Can you imagine reaching a trillion tokens context window in the future, as Sam speculated on Lex's podcast?

Re:Mixtral 8x22B by htrp on Wednesday April 17, 2024 @11:01AM (#40065667)
maybe we'll look back at token context windows like we look back at how much ram we have in a system.
Re:Mixtral 8x22B by frabjoused on Wednesday April 17, 2024 @11:04AM (#40065709)
I agree with this in the sense that once you have enough, you stop caring about the metric.
Re:Mixtral 8x22B by paradite on Wednesday April 17, 2024 @11:07AM (#40065763)
And how much RAM do you need to run Mixtral 8*22B? Probably not enough on a personal laptop.
Re:Mixtral 8x22B by user_7832 on Wednesday April 17, 2024 @11:13AM (#40065857)
Generally about ~1gb ram per billion parameters. I've run a 30b model (vicuna) on my 32gb laptop (but it was slow).
Re:Mixtral 8x22B by Lacerda69 on Wednesday April 17, 2024 @11:12AM (#40065839)
I run it fine on my 64gb RAM beast.
Re:Mixtral 8x22B by coder543 on Wednesday April 17, 2024 @11:16AM (#40065887)
At what quantization? 4-bit is 80GB. Less than 4-bit is rarely good enough at this point.
Re:Mixtral 8x22B by apexalpha on Wednesday April 17, 2024 @11:18AM (#40065927)
Is that normal ram of GPU ram?
Re:Mixtral 8x22B by samus on Wednesday April 17, 2024 @04:22PM (#40069671)
64GB is not GPU RAM, but system RAM. Consumer GPUs have 24GB at most, those with good value/price have way less. Current generation workstation GPUs are unaffordable; used can be found on ebay for a reasonable price, but they are quite slow. DDR5 RAM might be a better investment.
Re:Mixtral 8x22B by htrp on Wednesday April 17, 2024 @11:47AM (#40066338)
While there is a lot more HBM (or UMA if you're a Mac system) you need to run these LLM models, my overarching point is that at this point most systems don't have RAM constraints for most of the software you need to run and as a result, RAM becomes less of a selling point except in very specialized instances like graphic design or 3D rendering work.

If we have cheap billion token context windows, 99% of your use cases aren't going to hit anywhere close to that limit and as a result, your models will "just run"

Re:Mixtral 8x22B by bamboozled on Wednesday April 17, 2024 @11:20AM (#40065958)
I still don’t have enough RAM though ?
Re:Mixtral 8x22B by samus on Wednesday April 17, 2024 @04:29PM (#40069745)
RAM is simply too useful.
Re:Mixtral 8x22B by pseudosavant on Wednesday April 17, 2024 @12:23PM (#40066865)
FWIW, the 128k context window for GPT-4 is only for input. I believe the output content is still only 4k.
Re:Mixtral 8x22B by moffkalast on Wednesday April 17, 2024 @01:14PM (#40067538)
How does that make any sense on a decoder-only architecture?
Re:Mixtral 8x22B by grey8 on Wednesday April 17, 2024 @02:38PM (#40068552)
It's not about the model. The model can output more - it's about the API.

A better phrasing would be that they don't allow you to output more than 4k tokens per message.

Same with Anthropic and Claude, sadly.

Re:Mixtral 8x22B by ljhskyso on Thursday April 18, 2024 @02:49AM (#40073613)
you can always put the unfinished output as the input to continue forever until reaching the full 128k context window
Re:Mixtral 8x22B by creshal on Wednesday April 17, 2024 @11:30AM (#40066119)
Wasn't there a paper yesterday that turned context evaluation linear (instead of quadratic) and made effectively unlimited context windows possible? Between that and 1.58b quantization I feel like we're overdue for an LLM revolution.
Re:Mixtral 8x22B by samus on Wednesday April 17, 2024 @04:35PM (#40069794)
So far, people have come up with many alternatives for quadratic attention. Only recently have they proven their potential.
Re:Mixtral 8x22B by underlines on Wednesday April 17, 2024 @05:01PM (#40070022)
tons and tons of papers, most of them had some disadvantages. Can't have the cake and eat it too:

https://arxiv.org/html/2404.08801v1 Meta Megalodon

https://arxiv.org/html/2404.07143v1 Google Infini-Attention

https://arxiv.org/html/2402.13753v1 LongRoPE

and a ton more

Re:Mixtral 8x22B by afro88 on Wednesday April 17, 2024 @02:30PM (#40068464)
How useful is such a large input window when most of the middle isn't really used? I'm thinking mostly about coding. But wheb putting even say 20k tokens into the input, a good chunk doesn't seem to be "remembered" or used for the output
Re:Mixtral 8x22B by jakderrida on Wednesday April 17, 2024 @06:07PM (#40070568)
While you're 100% correct, they are working on ways to make the middle useful, such a "Needle in a Haystack" testing. When we say we wish for context length that large, I think it's implied we mean functionally. But you do make a really great point.