Re:Falcon 40B LLM (which beats Llama) now Apache 2.0
by amstan on Friday June 02, 2023 @03:09AM
(#36162045)
Woah, that's a cool direction. Thank you! I'll explore this.
[ Parent
| on HN ]
Re:Falcon 40B LLM (which beats Llama) now Apache 2.0
by washadjeffmad on Friday June 02, 2023 @07:14AM
(#36163658)
P40s are kind of a meme. Using ggmls has roughly the same performance at a fraction of the wattage on a dual-channel DDR5 system.
I still use GPTQ for 30B, but even CPU generates quickly enough at q5_1 on modern hardware.