Re:DeepSeek-R1 with Dynamic 1.58-bit Quantization
by danielhanchen on Tuesday January 28, 2025 @02:46PM
(#42857005)
The recommended temperature from DeepSeek is 0.6 so I leave it at that!
[ Parent
| on HN ]
Re:DeepSeek-R1 with Dynamic 1.58-bit Quantization
by smcleod on Tuesday January 28, 2025 @04:01PM
(#42857931)
I think most of the model creators share their model usage examples so high at 0.6-0.7 simply because it's what a lot of the client apps use. IMO this is WAY too high unless you're doing creative writing.
Generally I set temp to 0-0.4 at absolute most.
min_p actually needs a little temperature to work effectively so with min_p I almost always use 0.2
Re:DeepSeek-R1 with Dynamic 1.58-bit Quantization
by danielhanchen on Tuesday January 28, 2025 @04:15PM
(#42858119)
Ye lower temp is also good :) Tbh its all trial and error - I found temp=1.5, min_p=0.1 to be very useful for pass@k type workloads - ie calling the LLM multiple times and aggregating.
temp=0 is also good for singular outputs. For classification tasks, it's better to actually inspect the logits.
But my goto setting is always setting min_p at least 0.01 or 0.05! It vastly suppresses incorrect rare random tokens from being created, and it helps massively!