Re:Using GRPO to Beat o1, o3-mini and R1 at "Temporal Clue"
by bydgjohc on Thursday March 06, 2025 @03:29PM
(#43284780)
Any hypotheses on why the performance dropped suddenly while training?
[ Parent
| on HN ]
Re:Using GRPO to Beat o1, o3-mini and R1 at "Temporal Clue"
by a_wild_dandan on Thursday March 06, 2025 @03:55PM
(#43284988)
"We suspect the models may have converged too early on greedy strategies that worked out of the gate, but potentially limited their long-term prospects. A logical next step would explore approaches that encourage diverse responses, or that build capabilities incrementally (like with curriculum learning), or that assign larger rewards to particularly outstanding solutions incentivizing thorough exploration.
Additionally, we noted interesting patterns in output length during training. Initially, responses grew longer, then stabilized, before diverging near the end of training, with the 14B model’s responses getting longer and the 32B model’s response lengths collapsing, especially after reaching peak performance."
Re:Using GRPO to Beat o1, o3-mini and R1 at "Temporal Clue"
by bradhilton on Thursday March 06, 2025 @04:03PM
(#43285051)
Hi, other author here. I think the models converged on shallow/greedy strategies that improved performance up to a point, but are ultimately shortsighted, especially for harder puzzles.
Something interesting I noticed in the responses was that for shorter puzzles it would make deductions, building up a set additional "clues" for itself, before answering the question. However, for harder puzzles with more clues it would often merely repeat all the given clues and then try to directly answer the questions.
Maybe some form of curriculum learning would help, starting with easier puzzles and progressing to more challenging ones.
Other ideas to explore include:
- Distilling responses from stronger models - Encouraging exploration with entropy regularization or reward shaping - Training from base models instead of instruct models, like DeepSeek-R1-Zero
Re:Using GRPO to Beat o1, o3-mini and R1 at "Temporal Clue"
by kiratp on Thursday March 06, 2025 @11:08PM
(#43287332)
Is my understanding here correct? Could this be the reason?
Re:Using GRPO to Beat o1, o3-mini and R1 at "Temporal Clue"
by bradhilton on Thursday March 06, 2025 @04:25PM
(#43285213)
As for why they dropped suddenly, I don't really know. Sometimes models develop degenerate behaviors, but even when forking from the best checkpoint and lowering the learning rate or changing other hyperparameters, performance stills drops.
It's as if its fate has already been sealed many iterations ago.