News for hackers. Stuff that matters.
Re:Falcon 40B LLM (which beats Llama) now Apache 2.0 by esquire_900 on Friday June 02, 2023 @02:05AM (#36161633)
Some people use second hand P40 GPUs, which go for around 200-300$. Combine 3 of them with SLI and you've got 72GB of VRAM for less then $1000
[dead]
Re:Falcon 40B LLM (which beats Llama) now Apache 2.0 by esquire_900 on Thursday June 15, 2023 @05:37AM (#36338207)
http://nonint.com/ has some interesting posts about how he build a custom server to house 8 GPU's (3090's in this case). You're right that that will set you back more than $1000, though I was only referring to the GPU's themselves.
Re:Falcon 40B LLM (which beats Llama) now Apache 2.0 by amstan on Friday June 02, 2023 @03:09AM (#36162045)
Woah, that's a cool direction. Thank you! I'll explore this.
Re:Falcon 40B LLM (which beats Llama) now Apache 2.0 by washadjeffmad on Friday June 02, 2023 @07:14AM (#36163658)
P40s are kind of a meme. Using ggmls has roughly the same performance at a fraction of the wattage on a dual-channel DDR5 system.

I still use GPTQ for 30B, but even CPU generates quickly enough at q5_1 on modern hardware.