News for hackers. Stuff that matters.
Re:GPT-6 Sol and Luna by simonw on Tuesday September 22, 2026 @02:50PM (#49806237)
Yeah, it's interesting how much worse they are than the Astra pelicans. I think that reflects a tiny bit of genuine value still left in the benchmark, to be honest.
Re:GPT-6 Sol and Luna by hdz on Tuesday September 22, 2026 @03:08PM (#49806543)
Tons of value left, especially for open source models. I would say the benchmark is yet to be truly saturated (just look at the legs and seat to see what I am talking about) and I always look forward to seeing them. Thank you!
Re:GPT-6 Sol and Luna by Kotlopou on Tuesday September 22, 2026 @06:42PM (#49809298)
To me the main upshot of this benchmark is precisely that the pelicans still usually look a bit wonky. It's bizarre, since this definitely has a good solution, but it's in line with my experience that memorization of the training set just... isn't happening very much? As in, whether a model fails or not doesn't have much to do with whether that exact question was likely posed many times before.
Re:GPT-6 Sol and Luna by nomel on Tuesday September 22, 2026 @07:44PM (#49809794)
I think some additional value would be had by seeing how well it can modify the pelican.

Like, "now facing left", "sitting on the handlebars", or "with green spokes" to see if it can break out of some pretty obvious statistics in the training data!

And, there's always asking for an STL rather than an SVG!