this is a really interesting direction. i've been experimenting with “self-critique” style pipelines (plan → solve → critique) and they often help smaller models punch above their weight. the agent council idea is also appealing, although the cost/latency trade-off usually becomes the tricky part when multiple models run in parallel.

curious how often the judge actually disagrees with the first candidate answer in practice. does the council mostly refine reasoning, or does it sometimes lead to completely different conclusions?

Great question. Currently, we don’t track a formal disagreement rate between the judge and candidate 1, so I can’t give a reliable percentage yet. By design, the judge synthesizes advisor outputs and is instructed not to introduce new ideas, which usually leads to refinement rather than completely new conclusions. That said, advisors run independently and can produce different lines of reasoning, so the judge can still diverge from candidate 1 when another advisor’s argument is stronger. This keeps the final output grounded in the council’s collective input rather than the judge acting as a standalone model.

The Agent Council approach is interesting — having multiple small models debate in parallel and a judge synthesize feels like a more principled version of what people do manually when they cross-check answers between Claude, GPT, and Gemini. Curious whether the GSM8K gains hold up on less structured tasks where there isn't a single correct answer (e.g. summarization or open-ended reasoning).

That's a valid point. While the GSM8K gains show promise for structured reasoning, we're also curious to see how the council approach scales to more open-ended and less structured tasks like summarization. I'm planning to run some tests on those scenarios once I have a bit more free time to properly evaluate the results.