News for hackers. Stuff that matters.
Re:Mistral 7B, the most powerful language model for its size... by ComputerGuru on Wednesday September 27, 2023 @11:19AM (#37675923)
Are they? Unquantized, Llama 2 7b needs over 14GB of GPU (or shared) memory.
Re:Mistral 7B, the most powerful language model for its size... by polygamous_bat on Wednesday September 27, 2023 @11:27AM (#37676071)
"Unquantized" is the key word here: with quantization you can get a 4-8x improvement without much performance degradation.