Re:Ask HN: What's the best self hosted/local alternative to ...
by wing-_-nuts on Wednesday May 31, 2023 @10:42AM
(#36138975)
It irritates me to no end that people don't list the system requirements of various models.
How much ram and vram does one need to run 4,13,33,65B models at a reasonable speed?
edit: instead I'll ask this, what's the best model to run on a system with a 24gb 4090 and 64gb of ram?
Re:Ask HN: What's the best self hosted/local alternative to ...
by orost on Wednesday May 31, 2023 @10:58AM
(#36139126)
You can just barely fit a 33B GPTQ model in 24GB VRAM. It will be in 4-bit mode, and without maximum context size, but it will be quite fast. Or you can run from RAM+VRAM in GGML format with llama.cpp (or a derivative), which will easily fit 65B models even at 5 or 8 bits, but at much lower speed.
Re:Ask HN: What's the best self hosted/local alternative to ...
by evanchisholm on Wednesday May 31, 2023 @11:06AM
(#36139210)
It's fairly simple to estimate ram requirements based on parameter count:
https://blog.eleuther.ai/transformer-math/
Re:Ask HN: What's the best self hosted/local alternative to ...
by TheUninformed on Wednesday May 31, 2023 @12:14PM
(#36140064)
A 4-bit quantized 33B parameter model will fit on your GPU and you'll be able to use a 2048 token context too. (4-bit quantized larger models are better than smaller 8bit/16bit models)
You can run 4-bit quantized 65B models on your cpu, but it is slow, 1-2 tokens a second instead of 8-15 people typically get with a gpu, but you need two 24gb or an enterprise card with 48gb of ram to load them there.
https://old.reddit.com/r/LocalLLaMA/wiki/models has the information you are irritated about not being listed.
CPU Version: https://huggingface.co/TheBloke/guanaco-65B-GGML
GPU Version: https://huggingface.co/TheBloke/guanaco-65B-HF
4bit GPU Version: https://huggingface.co/TheBloke/guanaco-65B-GPTQ