In the end, medical students learn to become doctors. Cheaper models never learn to become more capable models, so the metaphor is not apt.
I compare this to OpenAI's symphony prompt which, at a high level, does exactly the same thing outlined here except model selection and doesn't really have the need for roles or hospital metaphors.
The comparison with OpenAI's Symphony is valid (the two systems do broadly the same thing). One thing that's different about Sinai is that it keeps the code author and reviewer in separate agents with separate context. At the time we built it, we suspected that this would lead to better outcomes, but again, we haven't validated that it does.
I confess that there's a lot more validation we could do with this model but we just haven't found the time. One thing we don't cover in the post is that this is a side project for both of us, so we have less time than we'd like to perform experimentation and validation.
I use a similar process for a local agent swarm approach (locally hosted Qwen-3.8-27b or Qwen-Flash-Next), used so far for personal-grade projects. Through tool and process accretion, the review stages are told to check both the work and reasoning of the implementation stage, via processing the pi.dev session log.
Through parsing the jsonl log, the reviewer sees the subagent prompt, the tool call sequence, and non-thinking narration along the way. That allows the reviewr to audit the implementer's process (e.g. were tests run?) and spot procedure violations or gross hallucinations. The reviewer also independently runs the test suite, so even a hallucinated pass is caught.
It's relatively expensive both in tokens and time spent running ideally duplicate tests, but the independent workflow has nonetheless caught errors that would very likely have been missed by a same session, same context review.
1. How does this work with greenfield lifts where the scope and final vision are not yet figured out?
2. Sorry if I missed this in the post, but will you open source this system?
What worked was a human writing the big-picture issue and the hospital decomposing it into a tree of smaller issues, which people could review with more rigor. We also did a few thing that didn't work; in one case we got a bit too over eager in trying to resolve a large slate of issues all at once, and left ourselves with a giant backlog of plans and code to review.
We're discussing internally what can be released in the open, but nothing to share yet.
Maybe trying this would be a good experiment? Say route 10 or 20% of any future issues to the “Chief” to troubleshoot and fix directly, and record those for further work?
I still find myself taking that approach fairly regularly. For most framework-level changes, or changes to core packages, I think the added scrutiny and deliberation is valuable. But for "leaf" components that don't have many dependents, using the hospital to move rapidly has been invaluable. I also encourage "old-fashioned" PRs for anyone who wants to gain a deeper understanding in a certain area of the code.
I like the control-group idea. It’s close to something we’re already building, where a human can label an issue to opt-in to write the plan or the code themselves. Routing a fixed % of issues that way would give us a baseline we don’t have right now.
> I also encourage "old-fashioned" PRs for anyone who wants to gain a deeper understanding in a certain area of the code.
Yeah, this is kind of what I meant. I think in some future, these are going to be really valuable for maintainers.
Would you say this is accurate? Have you had challenges generalizing to other sorts of work (you mentioned you're spreading it out to other repos)?