News for hackers. Stuff that matters.
Re:Falcon 40B LLM (which beats Llama) now Apache 2.0 by washadjeffmad on Friday June 02, 2023 @07:14AM (#36163658)
P40s are kind of a meme. Using ggmls has roughly the same performance at a fraction of the wattage on a dual-channel DDR5 system.

I still use GPTQ for 30B, but even CPU generates quickly enough at q5_1 on modern hardware.

There are no comments on this comment yet.