Re:Ask HN: What's the best self hosted/local alternative to ...
by orost on Wednesday May 31, 2023 @10:58AM
(#36139126)
You can just barely fit a 33B GPTQ model in 24GB VRAM. It will be in 4-bit mode, and without maximum context size, but it will be quite fast. Or you can run from RAM+VRAM in GGML format with llama.cpp (or a derivative), which will easily fit 65B models even at 5 or 8 bits, but at much lower speed.
Re:Ask HN: What's the best self hosted/local alternative to ...
by evanchisholm on Wednesday May 31, 2023 @11:06AM
(#36139210)
It's fairly simple to estimate ram requirements based on parameter count:
https://blog.eleuther.ai/transformer-math/
Re:Ask HN: What's the best self hosted/local alternative to ...
by TheUninformed on Wednesday May 31, 2023 @12:14PM
(#36140064)
A 4-bit quantized 33B parameter model will fit on your GPU and you'll be able to use a 2048 token context too. (4-bit quantized larger models are better than smaller 8bit/16bit models)
You can run 4-bit quantized 65B models on your cpu, but it is slow, 1-2 tokens a second instead of 8-15 people typically get with a gpu, but you need two 24gb or an enterprise card with 48gb of ram to load them there.
https://old.reddit.com/r/LocalLLaMA/wiki/models has the information you are irritated about not being listed.
How much ram and vram does one need to run 4,13,33,65B models at a reasonable speed?
edit: instead I'll ask this, what's the best model to run on a system with a 24gb 4090 and 64gb of ram?