this is a very valid point. I will expand on RL post train here.

I used GSPO because it worked really well for very long prompts (we're dealing with traces analyses so prompts are very very long, and by prompts I mean what goes into LLM call).

We RL-ed flow-1 in our harness, which made it very efficient at using tools it had. Our harness as I said in the post is basically coding agent, it has grep, read and couple of other aux tools. I spent a loooot of time and money on figuring out the right RL recipe and especially the reward. It was really hard to find the right balance between exploration and exploitation.

Some RL runs are still running, because the prod checkpoint as good as it is, thinks a lot. It produces a lot of thinking tokens, which affects the throughput, especially on the large, complex traces to be analyzed. So the next big challenge for me is to make it efficient thinker - i.e. reduce thinking tokens, without losing quality.

Have you tried SFT to guide it early to more efficient trajectories? I know it hurts generalization and I've seen folks claim that it hurts final performance, but for my limited budget I literally will not do rubric based RL without SFT to shape the thinking to the rubric anymore.

It's so much more compute efficient for my goals, though I do deal with subjective preference which makes exploration during training much harder

oh yeah, we did small-ish sft run to teach the model our trace compression style and so on. the problem with it is that you can't really teach good analysis this way, so RL was the core focus.