Re:GPT-OSS Reinforcement Learning
by Der_Einzige on Saturday September 27, 2025 @02:24AM
(#45393584)
You don't mention that MoE models horrible damage logprobs (by definition) and basically need a whole new theory of LLM sampling written for them.
Dense models are better for a reason, and the idea that "everyone is doing MoE now and dense models are dead" is total bunk nonsense.
You can quantize dense models, and 4 bit quantized Qwen 32B is still better than full precision GPT-OSS. Luckily Unsloth even gives you tools to go down to 1.58bits!
Dense models are better for a reason, and the idea that "everyone is doing MoE now and dense models are dead" is total bunk nonsense.
You can quantize dense models, and 4 bit quantized Qwen 32B is still better than full precision GPT-OSS. Luckily Unsloth even gives you tools to go down to 1.58bits!