Re:Ask HN: What's the best self hosted/local alternative to ...
by orost on Wednesday May 31, 2023 @10:58AM
(#36139126)
You can just barely fit a 33B GPTQ model in 24GB VRAM. It will be in 4-bit mode, and without maximum context size, but it will be quite fast. Or you can run from RAM+VRAM in GGML format with llama.cpp (or a derivative), which will easily fit 65B models even at 5 or 8 bits, but at much lower speed.
[ Parent
| on HN ]
There are no comments on this comment yet.