Re:Mistral 7B, the most powerful language model for its size...
by ComputerGuru on Wednesday September 27, 2023 @11:19AM
(#37675923)
Are they? Unquantized, Llama 2 7b needs over 14GB of GPU (or shared) memory.
[ Parent
| on HN ]
Re:Mistral 7B, the most powerful language model for its size...
by polygamous_bat on Wednesday September 27, 2023 @11:27AM
(#37676071)
"Unquantized" is the key word here: with quantization you can get a 4-8x improvement without much performance degradation.