News for hackers. Stuff that matters.
Re:Perplexity Labs Playground by avereveard on Tuesday December 19, 2023 @04:04PM (#38701517)
with llama.cpp and a 12gb 3060 they can get the an entire mistral model at Q5_K_M n ram with the full 32k context. I recommend openhermes-2.5-mistral-7b-16k with USER: ASSISTANT: instructions, it's working surprisingly well for content production (let's say everything except logic and math, but that's not the strong suite of 7b models in general)

There are no comments on this comment yet.