Re:ChatGPT's 1-Year Anniversary: Are Open-Source Large Langu...
by tarruda on Friday December 01, 2023 @05:37AM
(#38485298)
I get 10 tokens/second on a 4-bit 13B model with 8GB VRAM offloading as much as possible to the GPU. At this speed, I cannot read the LLM output as fast as it generates, so I consider it to be sufficient.
[ Parent
| on HN ]
Re:ChatGPT's 1-Year Anniversary: Are Open-Source Large Langu...
by johnbellone on Friday December 01, 2023 @10:43AM
(#38488045)
Which video card?