News for hackers. Stuff that matters.
Re:ChatGPT's 1-Year Anniversary: Are Open-Source Large Langu... by tarruda on Friday December 01, 2023 @05:37AM (#38485298)
I get 10 tokens/second on a 4-bit 13B model with 8GB VRAM offloading as much as possible to the GPU. At this speed, I cannot read the LLM output as fast as it generates, so I consider it to be sufficient.
Re:ChatGPT's 1-Year Anniversary: Are Open-Source Large Langu... by johnbellone on Friday December 01, 2023 @10:43AM (#38488045)
Which video card?
Re:ChatGPT's 1-Year Anniversary: Are Open-Source Large Langu... by tarruda on Friday December 01, 2023 @11:06AM (#38488290)
RTX 3070 Max-Q (laptop)