Re:Show HN: RULER – Easily apply RL to any agent
by maxrmk on Friday July 11, 2025 @07:10PM
(#44537726)
Very cool. Do you do anything to mitigate ordering bias in the evaluation function, or do you just expect it to average out over time?
[ Parent
| on HN ]
Re:Show HN: RULER – Easily apply RL to any agent
by kcorbitt on Friday July 11, 2025 @07:40PM
(#44537925)
No, we don't do anything. Theoretically we could judge several times with different ordering.
We could measure order bias really easily though; we just need to look at the average score by rollout position across many runs. I'll add that to my list of experiments!