News for hackers. Stuff that matters.
Re:Mistral 7B, the most powerful language model for its size... by yieldcrv on Wednesday September 27, 2023 @11:07AM (#37675724)
what do you use "the most powerful language model for its size" for?
Re:Mistral 7B, the most powerful language model for its size... by ShrigmaMale on Wednesday September 27, 2023 @11:10AM (#37675778)
Probably a better candidate for local applications? Guessing this was a trial balloon against larger models coming though.
Re:Mistral 7B, the most powerful language model for its size... by linkpuff on Wednesday September 27, 2023 @11:12AM (#37675819)
According to their website, it's "Optimal for: low latency, text summarisation, classification, text completion, code completion"
Re:Mistral 7B, the most powerful language model for its size... by snowram on Wednesday September 27, 2023 @11:11AM (#37675798)
7B models are small enough to be usable on a smartphone, so a local handheld assistant sounds like a use case.
Re:Mistral 7B, the most powerful language model for its size... by ComputerGuru on Wednesday September 27, 2023 @11:19AM (#37675923)
Are they? Unquantized, Llama 2 7b needs over 14GB of GPU (or shared) memory.
Re:Mistral 7B, the most powerful language model for its size... by polygamous_bat on Wednesday September 27, 2023 @11:27AM (#37676071)
"Unquantized" is the key word here: with quantization you can get a 4-8x improvement without much performance degradation.
Re:Mistral 7B, the most powerful language model for its size... by minimaxir on Wednesday September 27, 2023 @12:24PM (#37677065)
"usable" is not the same as practical.

Even running a quantized and optimized LLM on a smartphone would kill battery life at minimum.

Re:Mistral 7B, the most powerful language model for its size... by brucethemoose2 on Wednesday September 27, 2023 @12:52PM (#37677509)
Try MLC-LLM. Its not as bad as you'd think.

In the future(?), they will probably use the AI blocks instead of the GPU, which are very low power.

Re:Mistral 7B, the most powerful language model for its size... by kirill5pol on Wednesday September 27, 2023 @01:33PM (#37678136)
One big one is speculative decoding for larger models, the more “accurate”* the smaller model, the more the speed up on the bigger model

* as in matches the token that the larger model would output

Re:Mistral 7B, the most powerful language model for its size... by sroussey on Wednesday September 27, 2023 @01:38PM (#37678208)
It would be better if they also had a 70B model for this.

They would need the same vocabulary, etc. What else?