Last time I did this I found it quite hard to come up with tasks that have clear evaluation outcomes
Getting the framework right isn't that hard, but requires some careful thought. You probably don't want to just checkout a repo at a certain commit hash - agents will explore other commits and branches. (Public repos are also risky in general: agents with internet access can find later commits, and public repos could be in the training data anyway.) I use `git archive` and give the runner just the files at that commit and nothing else. I also do the run inside an isolated container that is recreated each run so it can't find e.g. related files left in /tmp from a previous run.
Getting the benchmark tasks right is harder. I don't think there's any way to get around writing the task and its evaluation by hand. I really tried to get a few different frontier LLMs to do it and they're all terrible at "imagining" what an agent would need to solve a problem without giving away the answer or providing too little information. Also quite bad at writing evaluation criteria that tests whether the solution matches the assignment (versus just how close it is to a golden solution, or how close it is to how they would solve it)
I personally don't like or trust using LLM as a grader either, but I guess people manage to get that to work well enough.
It's a lot more imprecise than the public benchmarks, but it's tailored to my own work and definitely isn't being benchmaxxed by the model developers. I was able to tell really quickly that Opus 5.5 delivered on the hype, and that some other releases (a couple of the more recent open releases, and pre-Carbon gemini models) were not as good for me as public benchmarks made them look.
(no results for either 6.2 yet, will take another ~6-8 hours, but I encourage people reading this with spare tokens to just make their own like I did instead of listen to me)