News for hackers. Stuff that matters.
Re:GPT‑6.2 Astra and Sol by eli on Tuesday November 17, 2026 @02:52PM (#50537175)
I have built my own benchmarks and it's very valuable to have them but I'm skeptical you can get good results vibe coding them.

Getting the framework right isn't that hard, but requires some careful thought. You probably don't want to just checkout a repo at a certain commit hash - agents will explore other commits and branches. (Public repos are also risky in general: agents with internet access can find later commits, and public repos could be in the training data anyway.) I use `git archive` and give the runner just the files at that commit and nothing else. I also do the run inside an isolated container that is recreated each run so it can't find e.g. related files left in /tmp from a previous run.

Getting the benchmark tasks right is harder. I don't think there's any way to get around writing the task and its evaluation by hand. I really tried to get a few different frontier LLMs to do it and they're all terrible at "imagining" what an agent would need to solve a problem without giving away the answer or providing too little information. Also quite bad at writing evaluation criteria that tests whether the solution matches the assignment (versus just how close it is to a golden solution, or how close it is to how they would solve it)

I personally don't like or trust using LLM as a grader either, but I guess people manage to get that to work well enough.

There are no comments on this comment yet.