Re:Perplexity Labs Playground
by M4v3R on Tuesday December 19, 2023 @01:59PM
(#38699930)
You need to pick the correct model size and quantization for the amount of GPU RAM you have. For any given model don’t download the default file, instead go to Tags section on Ollama’s page and pick a quantization whose size in GB is at most 2/3rd of your available RAM, and it should work. For example in your case Mistral-7B q4_0 and even q8_0 should work perfectly.
[ Parent
| on HN ]
whats the intuition for 2/3 of RAM?
Re:Perplexity Labs Playground
by avereveard on Tuesday December 19, 2023 @04:02PM
(#38701495)
you need some leftover for holding the context
Because there’s always some overhead during inference plus you don’t want to fill all your available RAM because you risk swapping to disk which will make everything slow to a crawl.