News for hackers. Stuff that matters.
Re:GPT‑6.2 Astra and Sol by mlinsey on Tuesday November 17, 2026 @01:30PM (#50535975)
If you have the quota or budget for it, I recommend vibe-coding your own personal benchmark. I have a suite of 8 projects that I made with models this year, and replay them - give the new model a checkout of the repo at that point, and the project spec. For the design task, I have it throw up a blind voting page for me to quickly judge, and for implementation I have a review agent do a general score and also hunt for problems/edge cases that I really found.

It's a lot more imprecise than the public benchmarks, but it's tailored to my own work and definitely isn't being benchmaxxed by the model developers. I was able to tell really quickly that Opus 5.5 delivered on the hype, and that some other releases (a couple of the more recent open releases, and pre-Carbon gemini models) were not as good for me as public benchmarks made them look.

(no results for either 6.2 yet, will take another ~6-8 hours, but I encourage people reading this with spare tokens to just make their own like I did instead of listen to me)

Re:GPT‑6.2 Astra and Sol by grey-area on Tuesday November 17, 2026 @01:33PM (#50536020)
This is much more interesting IMO than any public benchmark. They are definitely targeting those benchmarks and model capability does seem to have plateaued somewhat.
Re:GPT‑6.2 Astra and Sol by srcreigh on Tuesday November 17, 2026 @01:36PM (#50536061)
please update !
Re:GPT‑6.2 Astra and Sol by Havoc on Tuesday November 17, 2026 @01:38PM (#50536090)
What type of tasks did you use for this? Evaluation?

Last time I did this I found it quite hard to come up with tasks that have clear evaluation outcomes

Re:GPT‑6.2 Astra and Sol by bertmuir on Tuesday November 17, 2026 @01:54PM (#50536366)
[delayed]
Re:GPT‑6.2 Astra and Sol by khimaros on Tuesday November 17, 2026 @02:37PM (#50536984)
Re:GPT‑6.2 Astra and Sol by seizethecheese on Tuesday November 17, 2026 @01:54PM (#50536369)
[delayed]
Re:GPT‑6.2 Astra and Sol by behat on Tuesday November 17, 2026 @02:03PM (#50536490)
This is great, and I have been setting up infra to do this very thing. How many sessions to you replay typically to get a reasonably confident estimate? Assuming each project in your suite was built over multiple sessions.
Re:GPT‑6.2 Astra and Sol by eli on Tuesday November 17, 2026 @02:52PM (#50537175)
I have built my own benchmarks and it's very valuable to have them but I'm skeptical you can get good results vibe coding them.

Getting the framework right isn't that hard, but requires some careful thought. You probably don't want to just checkout a repo at a certain commit hash - agents will explore other commits and branches. (Public repos are also risky in general: agents with internet access can find later commits, and public repos could be in the training data anyway.) I use `git archive` and give the runner just the files at that commit and nothing else. I also do the run inside an isolated container that is recreated each run so it can't find e.g. related files left in /tmp from a previous run.

Getting the benchmark tasks right is harder. I don't think there's any way to get around writing the task and its evaluation by hand. I really tried to get a few different frontier LLMs to do it and they're all terrible at "imagining" what an agent would need to solve a problem without giving away the answer or providing too little information. Also quite bad at writing evaluation criteria that tests whether the solution matches the assignment (versus just how close it is to a golden solution, or how close it is to how they would solve it)

I personally don't like or trust using LLM as a grader either, but I guess people manage to get that to work well enough.