Re:ChatGPT's 1-Year Anniversary: Are Open-Source Large Langu...
by tarruda on Friday December 01, 2023 @05:37AM
(#38485298)
I get 10 tokens/second on a 4-bit 13B model with 8GB VRAM offloading as much as possible to the GPU. At this speed, I cannot read the LLM output as fast as it generates, so I consider it to be sufficient.
Re:ChatGPT's 1-Year Anniversary: Are Open-Source Large Langu...
by johnbellone on Friday December 01, 2023 @10:43AM
(#38488045)
Which video card?
It's also possible to run on CPU from system RAM, to split the workload across GPU and CPU, or even from a memory-mapped file on disk. Some people have posted benchmarks online [1] and naturally, the faster your RAM and CPU the better.
My personal experience is running from CPU/system ram is painfully slow. But that's partly because I only experimented with models that were too big to fit on my GPU, so part of the slowness is due to their large size.
[1] https://www.reddit.com/r/LocalLLaMA/comments/14ilo0t/extensi...