News for hackers. Stuff that matters.
Re:ChatGPT's 1-Year Anniversary: Are Open-Source Large Langu... by michaelt on Friday December 01, 2023 @05:16AM (#38485171)
If your GPU has ~16GB of vram, you can run a 13B model in "Q4_K_M.gguf" format and it'll be fast. Maybe even ~12GB.

It's also possible to run on CPU from system RAM, to split the workload across GPU and CPU, or even from a memory-mapped file on disk. Some people have posted benchmarks online [1] and naturally, the faster your RAM and CPU the better.

My personal experience is running from CPU/system ram is painfully slow. But that's partly because I only experimented with models that were too big to fit on my GPU, so part of the slowness is due to their large size.

[1] https://www.reddit.com/r/LocalLLaMA/comments/14ilo0t/extensi...

Re:ChatGPT's 1-Year Anniversary: Are Open-Source Large Langu... by tarruda on Friday December 01, 2023 @05:37AM (#38485298)
I get 10 tokens/second on a 4-bit 13B model with 8GB VRAM offloading as much as possible to the GPU. At this speed, I cannot read the LLM output as fast as it generates, so I consider it to be sufficient.
Re:ChatGPT's 1-Year Anniversary: Are Open-Source Large Langu... by johnbellone on Friday December 01, 2023 @10:43AM (#38488045)
Which video card?
Re:ChatGPT's 1-Year Anniversary: Are Open-Source Large Langu... by tarruda on Friday December 01, 2023 @11:06AM (#38488290)
RTX 3070 Max-Q (laptop)
Re:ChatGPT's 1-Year Anniversary: Are Open-Source Large Langu... by gardnr on Friday December 01, 2023 @06:38AM (#38485662)
I can fit 13B Q4 K M models on a 12GB RTX 3060. It OOMs when the context window goes above 3k. I get 25 tok/s.