> I find coding to be harder to benchmark because there are so many ways to write the same solution. A "correct" solution may be terrible in another context due to loss of speed, security, etc.
Yeah, I also do my own benchmarks, but as you said, coding ones are a bit harder. Currently I'm mostly benchmarking the accuracy of the tools I've written, which do bite-sized work. One tool is for editing parts of files, another to rewrite files fully, and so on, and each is individually benchmarked. But they're very specific, I do no overall benchmark for "Change feature X to do Y" which would span a "session", haven't find any good way of evaluating the results, just like you :)
Re:GPT-5
by anshumankmr on Thursday August 07, 2025 @09:04PM
(#44832259)
Here’s one of my favourite questions. Every single
Model prior to this including 4o, o3 use to fumble this.
https://photos.app.goo.gl/zzfKKqDYFtMAP9Vb8
I am yet to test this out end to end
I find coding to be harder to benchmark because there are so many ways to write the same solution. A "correct" solution may be terrible in another context due to loss of speed, security, etc.