Re:Mistral 7B, the most powerful language model for its size...
by snowram on Wednesday September 27, 2023 @11:11AM
(#37675798)
7B models are small enough to be usable on a smartphone, so a local handheld assistant sounds like a use case.
[ Parent
| on HN ]
Re:Mistral 7B, the most powerful language model for its size...
by ComputerGuru on Wednesday September 27, 2023 @11:19AM
(#37675923)
Are they? Unquantized, Llama 2 7b needs over 14GB of GPU (or shared) memory.
Re:Mistral 7B, the most powerful language model for its size...
by polygamous_bat on Wednesday September 27, 2023 @11:27AM
(#37676071)
"Unquantized" is the key word here: with quantization you can get a 4-8x improvement without much performance degradation.
Re:Mistral 7B, the most powerful language model for its size...
by minimaxir on Wednesday September 27, 2023 @12:24PM
(#37677065)
"usable" is not the same as practical.
Even running a quantized and optimized LLM on a smartphone would kill battery life at minimum.
Re:Mistral 7B, the most powerful language model for its size...
by brucethemoose2 on Wednesday September 27, 2023 @12:52PM
(#37677509)
Try MLC-LLM. Its not as bad as you'd think.
In the future(?), they will probably use the AI blocks instead of the GPU, which are very low power.