Re:LLaMA Now Goes Faster on CPUs
by astrange on Monday April 01, 2024 @02:24AM
(#39891365)
If you want factual answers from a local model it might help to turn the temperature down.
[ Parent
| on HN ]
It would also help if I had more VRAM and wasn't running a 7B parameter 4-bit quantized model.
> If you want factual answers from a local model it might help to turn the temperature down.
This makes sense. If you interact with a language model and it says something wrong it is your fault
You're not "interacting with a language model", you're running a program (llama.cpp) with a sampling algorithm which is not set to maximum factualness by default.
It's like how you have to set x264 to the anime tuning or the film tuning depending on what you run it on.