News for hackers. Stuff that matters.
Re:GPT-5 for Developers by aliljet on Thursday August 07, 2025 @01:54PM (#44827943)
Between Opus aand GPT-5, it's not clear there's a substantial difference in software development expertise. The metric that I can't seem to get past in my attempts to use the systems is context awareness over long-running tasks. Producing a very complex, context-exceeding objective is a daily (maybe hourly) ocurrence for me. All I care about is how these systems manage context and stay on track over extended periods of time.

What eval is tracking that? It seems like it's potentially the most imporatnt metric for real-world software engineering and not one-shot vibe prayers.

Re:GPT-5 for Developers by abossy on Thursday August 07, 2025 @03:50PM (#44829492)
At my company (Charlie Labs), we've had a tremendous amount of success with context awareness over long-running tasks with GPT-5 since getting access a few weeks ago. We ran an eval to solve 10 real Github issues so that we could measure this against Claude Code and the differences were surprisingly large. You can see our write-up here:

https://charlielabs.ai/research/gpt-5

Often, our tasks take 30-45 minutes and can handle massive context threads in Linear or Github without getting tripped up by things like changes in direction part of the way through the thread.

While 10 issues isn't crazy comprehensive, we found it to be directionally very impressive and we'll likely build upon it to better understand performance going forward.

Re:GPT-5 for Developers by bartman on Thursday August 07, 2025 @05:03PM (#44830334)
I am not (usually) photosensitive, but the animated static noise on your websites causes noticable flickering on various screens I use and made it impossible for me to read your article.

For better accessibility and a safer experience[1] I would recommend not animating the background, or at least making it easily togglable.

[1] https://developer.mozilla.org/en-US/docs/Web/Accessibility/G...

Re:GPT-5 for Developers by neom on Thursday August 07, 2025 @05:19PM (#44830499)
Removed- sorry, and thank you for the feedback.
Re:GPT-5 for Developers by pxc on Thursday August 07, 2025 @05:38PM (#44830697)
Love your responsiveness here!

Edited to add: I am, in fact, photosensitive (due to a genetic retinal condition), and for my eyes, your site as it is very easy to read, and the visualizations look great.

Re:GPT-5 for Developers by bartman on Thursday August 07, 2025 @06:43PM (#44831311)
Thank you!

Love that you included the judge prompts in your article.

Re:GPT-5 for Developers by neom on Thursday August 07, 2025 @06:49PM (#44831368)
Please let me know what you would like to see more of. Evals are something we take serious, I think this post was ok enough given our constraints, but I'd like to produce content people find useful and I think we can do a lot better.
Re:GPT-5 for Developers by jeanlucas on Thursday August 07, 2025 @05:34PM (#44830672)
Nice,
Re:GPT-5 for Developers by MPSFounder on Thursday August 07, 2025 @05:15PM (#44830460)
I concur. Awful UI
Re:GPT-5 for Developers by RyanHamilton on Friday August 08, 2025 @03:56AM (#44834631)
Did you sign any kind of agreement with a non disparagement clause to get early access? I'm asking because if you did, your data point isn't useful. It would mean anyone else that tried it and got worse results wouldn't be able to post here. We would just be seeing the successful data points.
Re:GPT-5 for Developers by htrp on Friday August 08, 2025 @08:39AM (#44836307)
Even if they didn't, overly critical or negative commentary will mean their removal from the list of trusted testers
Re:GPT-5 for Developers by neom on Friday August 08, 2025 @08:47AM (#44836367)
They didn't say anything to us, nothing was approved, just eng <> eng discussion about the model. Also nothing was cherry picked etc etc - I don't care what OAI thinks, I care about producing the best product and showing you our findings.
Re:GPT-5 for Developers by TechDebtDevin on Friday August 08, 2025 @09:19AM (#44836628)
Waitig 30-45 minutes for code, that you're still going to have to read from top to bottom to make sure it doesn't have anything dumb in it, does not seem like a productivity enhancement. I would quit If I was an engineer and told to do this.
Re:GPT-5 for Developers by rantallion on Friday August 08, 2025 @10:56AM (#44837774)
If you're doing nothing in that 30-45 minutes other than stare at a loading screen, you're doing it wrong.

I'm not sold on the efficacy of AI and I share your reservations about having to scrutinise their output, but I see great value in being able to offload a long-running task to someone/something else and only have to check back later. In the meantime, I can be doing something else - like sitting in those planning meetings we all enjoy!

Re:GPT-5 for Developers by abossy on Friday August 08, 2025 @11:54AM (#44838485)
I love sitting in those planning meetings, too. /s

This is exactly right. We've adapted our workflow to kick off a task and then kick off the next one and the next. Then we review the work of each as they come through. It's just CPU pipelining for human workflow.

The process is far from perfect but the throughput is very high. The limiting factor is review. I spend most of my time doing line-by-line review of AI output and asking questions about things I'm unsure of. It's a very different job from the way I historically operated, which involved tight code -> verify loops of manually written code.

Re:GPT-5 for Developers by RobinL on Thursday August 07, 2025 @03:26PM (#44829211)
Totally agree. At the moment I find that frontier LLMs are able to solve most of the problems I throw at them given enough context. Most of my time is spent working out what context they're missing when they fail. So the thing that would help me most is much a much more focussed ability to gather context.

For my use cases, this is mostly needing to be really home in on relevant code files, issues, discussions, PRs. I'm hopeful that GPT5 will be a step forward in this regard that isn't fully captured in the benchmark results. It's certainly promising that it can achieve similar results more cheaply than e.g. Opus.

Re:GPT-5 for Developers by swader999 on Thursday August 07, 2025 @02:01PM (#44828061)
If GPT 5 truly has 400k context, that might be all it needs to meaningfully surpass Opus.
Re:GPT-5 for Developers by andrewmutz on Thursday August 07, 2025 @02:39PM (#44828645)
Having a large context window is very different from being able to effectively use a lot of context.

To get great results, it's still very important to manage context well. It doesn't matter if the model allows a very large context window, you can't just throw in the kitchen sink and expect good results

Re:GPT-5 for Developers by dimal on Thursday August 07, 2025 @02:34PM (#44828564)
Even with large contexts there's diminishing returns. Just having the ability to stuff more tokens in context doesn't mean the model can effectively use it. As far as I can tell, they always reach a point in which more information makes things worse.
Re:GPT-5 for Developers by Byamarro on Thursday August 07, 2025 @02:39PM (#44828640)
More of a question is its context rot tendency than the size of its context :) LLMs are supposed to load 3 bibles into their context, but they forget what they were about to do after loading a 600LoC of locales.
Re:GPT-5 for Developers by simonw on Thursday August 07, 2025 @02:22PM (#44828404)
It's 272,000 input tokens and 128,000 output tokens.
Re:GPT-5 for Developers by zurfer on Thursday August 07, 2025 @04:40PM (#44830056)
Woah that's really kind of hidden. But I think you can specify max output tokens. Need to test that!
Re:GPT-5 for Developers by 6thbit on Thursday August 07, 2025 @05:24PM (#44830551)
Oh, I had not grasped that the “context window” size advertised had to include both input and output.

But is it really 272k even if the output was say 10k? Cause it does say “max output” in the docs, so I wonder

Re:GPT-5 for Developers by simonw on Thursday August 07, 2025 @06:25PM (#44831149)
This is the only model where the input limit and the context limit are different values. OpenAI docs team are working on updating that page.
Re:GPT-5 for Developers by dudeinhawaii on Friday August 08, 2025 @10:24AM (#44837351)
The website clearly lays them out as 400k input and 128k output [1]. I just updated my AI apps to support the new models. I routinely fill the entire context on large code calls. Input is not a "shared" context.

I found 100k was barely enough for a single project without spillover, so 4x allows for linking more adjacent codebases for large scale analysis.

[1] https://platform.openai.com/docs/models/gpt-5

Re:GPT-5 for Developers by AS04 on Thursday August 07, 2025 @02:08PM (#44828176)
400k context with 100% on the fiction livebench would make GPT-5 the undisputably best model IMHO. Don't think it will achieve that though, sadly.
Re:GPT-5 for Developers by tekacs on Thursday August 07, 2025 @02:48PM (#44828756)
Coupled with the humungous price difference...
Re:GPT-5 for Developers by nadis on Thursday August 07, 2025 @03:17PM (#44829096)
It's pretty vague, but the OP had this callout:

>"GPT‑5 is the strongest coding model we’ve ever released. It outperforms o3 across coding benchmarks and real-world use cases, and has been fine-tuned to shine in agentic coding products like Cursor, Windsurf, GitHub Copilot, and Codex CLI. GPT‑5 impressed our alpha testers, setting records on many of their private internal evals."

Re:GPT-5 for Developers by logicchains on Thursday August 07, 2025 @03:00PM (#44828895)
>Between Opus aand GPT-5, it's not clear there's a substantial difference in software development expertise.

If there's no substantial difference in software development expertise then GPT-5 absolutely blows Opus out of the water due to being almost 10x cheaper.

Re:GPT-5 for Developers by spiderice on Thursday August 07, 2025 @05:26PM (#44830571)
Does OpenAI provide a $200/month option that lets me use as much GPT-5 I want inside of Codex?

Because if not, I'd still go with Opus + Claude Code. I'd rather be able to tell my employer, "this will cost you $200/month" than "this might cost you less than $200/month, but we really don't know because it's based on usage"

Re:GPT-5 for Developers by konarkm on Thursday August 07, 2025 @07:51PM (#44831825)
The ChatGPT paid subscriptions now come with Codex CLI usage included
Re:GPT-5 for Developers by t1amat on Friday August 08, 2025 @12:52AM (#44833553)
Is this actually true? Last I checked (a week ago?) Codex the agents were free at some tiers in a preview capacity (with future rate limits based on tier), but codex cli was not. With codex cli you can log in but the purpose of that is to link it to an API key where you pay per use. The sub tiers give one time credits you would burn through quickly.
Re:GPT-5 for Developers by Deradon on Friday August 08, 2025 @01:20AM (#44833713)
Found this in the GPT-5 Announcement:

> Availability and access > GPT‑5 is starting to roll out today to all Plus, Pro, Team, and Free users, with access for Enterprise and Edu coming in one week. Pro, Plus, and Team users can also start coding with GPT‑5 in the Codex CLI (opens in a new window) by signing in with ChatGPT.

Re:GPT-5 for Developers by mh- on Thursday August 07, 2025 @07:21PM (#44831611)
To be clear, Claude doesn't provide that either. You can get "usage limited" off of Opus on the $200/mo plan.
Re:GPT-5 for Developers by mrheosuper on Thursday August 07, 2025 @10:32PM (#44832804)
Does Claude ?
Re:GPT-5 for Developers by user3939382 on Thursday August 07, 2025 @08:54PM (#44832204)
I just asked codex to copy a file and it took almost a minute to think about it and cost $0.05. This is something Claude Code would have done in seconds.
Re:GPT-5 for Developers by realusername on Thursday August 07, 2025 @01:56PM (#44827979)
Personally I think I'll wait for another 10x improvement for coding because with the current way it's going, they clearly need that.
Re:GPT-5 for Developers by fsloth on Thursday August 07, 2025 @02:10PM (#44828206)
From my experience when used through IDE such as Cursor the current gen Claude model enables impressive speedruns over commodity tasks. My context is a CAD application I’ve been writing as a hobby. I used to work in that field for a decade so have a pretty good touch on how long I would expect tasks to take. I’m using mostly a similar software stack as that at previous job and am definetly getting stuff done much faster on holiday at home than at that previous work. Of course the codebase is also a lot smaller, intrinsic motivation, etc, but still.
Re:GPT-5 for Developers by realusername on Thursday August 07, 2025 @03:28PM (#44829227)
I've done pretty much the same as you (Cursor/Claude) for our large Rails/React codebase at work and the experience has been horrific so far, I reverted back to vscode.
Re:GPT-5 for Developers by fsloth on Friday August 08, 2025 @03:39AM (#44834553)
Yeah! It's quite possible my scenario is in the "happy accident" valley.

I'm using it mostly for C#, WPF and OpenTK. The type system seems to help a lot.

The UI logic it recommends is mostly god awful. But at least for me when it's given a pattern it can apply, it does so pretty well.

Re:GPT-5 for Developers by 42lux on Thursday August 07, 2025 @02:25PM (#44828436)
How often do you have to build the simple scaffolding though?
Re:GPT-5 for Developers by fsloth on Friday August 08, 2025 @03:37AM (#44834540)
At a real job? Not that often! And it's miserable in large scale architecture.

However, at leas for me there is lots of "small enough context" boilerplate that the context can deal with.

Clearly this is not a tool in the sense it's predictable.

Re:GPT-5 for Developers by bdangubic on Thursday August 07, 2025 @01:58PM (#44828007)
context awareness over long-running tasks

don’t have long-running tasks, llms or not. break the problem down into small manageable chunks and then assemble it. neither humans nor llms are good at long-running tasks.

Re:GPT-5 for Developers by bastawhiz on Thursday August 07, 2025 @02:24PM (#44828424)
> neither humans nor llms are good at long-running tasks.

That's a wild comparison to make. I can easily work for an hour. Cursor can hardly work for a continuous pomodoro. "Long-running" is not a fixed size.

Re:GPT-5 for Developers by bdangubic on Thursday August 07, 2025 @03:43PM (#44829401)
I just finished my workday, 8hrs with Claude Code. No single task took more than 20 minutes total. Cleared context after each task and asked it to summarize for itself the previous task before I cleared context. If I ran this as a continuous 8hr task it would have died after 35-ish minutes. Just know the limitations (like with any other tool) and you’ll be good :)
Re:GPT-5 for Developers by 0x457 on Thursday August 07, 2025 @04:00PM (#44829626)
I always find it wild that none of these tools use VCS - completed logical unit of work, make a commit, drop entire context related to that commit, while referring commit, continue onto the next stage, rinse and repeat.

Claud always misunderstands how API exported by my service works and every compaction it forgets all over and commits "oh api has change since last time I've used, let me use different query parameters", my brother Christ nothing has changed, and you are the one who made this API.

Re:GPT-5 for Developers by bastawhiz on Thursday August 07, 2025 @06:31PM (#44831205)
You can use cursor rules to tell cursor to update the project cursor rules with details about the API.
Re:GPT-5 for Developers by bdangubic on Thursday August 07, 2025 @08:16PM (#44831982)
that would mean someone actually tried to learn how to use it :)
Re:GPT-5 for Developers by 0x457 on Friday August 08, 2025 @12:42PM (#44839046)
Yes, I can, and I do, I'm pointing out that compressing an entire conversation history into a single message is so lossy that I might as well start a new session.

Yes, I can also tell any agent to commit more often, but that's again not what I'm saying. I'm saying version control can be integrated way deeper into agent workflow.

Re:GPT-5 for Developers by bahmboo on Thursday August 07, 2025 @06:48PM (#44831357)
Roo Code does this
Re:GPT-5 for Developers by bdangubic on Thursday August 07, 2025 @09:26PM (#44832402)
I do exactly this - except I want control to define logical units of work
Re:GPT-5 for Developers by 0x457 on Friday August 08, 2025 @12:45PM (#44839087)
That's how Kiro works and how the agent that I'm working on works.

Kiro creates tasks from spec document, and you can revise tasks either by prompting or editing tasks.md file.

My thing works a bit differently, but essentially the same.

Re:GPT-5 for Developers by jondwillis on Friday August 08, 2025 @10:29AM (#44837430)
Idk I had cursor/claude untangle and commit to two separate logical branches yesterday from a bunch of random working copy changes that I had made. You can prompt it to use git commands and it works well enough in my experience.
Re:GPT-5 for Developers by 0x457 on Friday August 08, 2025 @12:39PM (#44839018)
Yes, you can prompt, I find it confusing why that + worktrees (i.e. agent working on its own working copy of the repo) is the standard/
Re:GPT-5 for Developers by echelon on Thursday August 07, 2025 @02:40PM (#44828654)
Humans can error correct.

LLMs multiply errors over time.

Re:GPT-5 for Developers by novok on Thursday August 07, 2025 @06:15PM (#44831048)
I think that is because you do implicit plan tracking, creation and modification of the plan in your head in light of new information and then follow that plan. I'm not sure these tools do that very well.

The long running task, at it's core, is composed of many smaller tasks and you mostly focus on one task at a time per brain part. It's why you cannot read two streams of text simultaneously even if both are in your visual focus field.

Re:GPT-5 for Developers by bdangubic on Thursday August 07, 2025 @09:25PM (#44832396)
you making too much sense :)
Re:GPT-5 for Developers by raducu on Friday August 08, 2025 @02:40AM (#44834187)
> you do implicit plan tracking, creation and modification of the plan in your head in light of new information and then follow that plan. I'm not sure these tools do that very well.

I think the plan is not just words, if it was, you could read a book on how to ride a bike.

Because we communicate in language and because code output is also a language we think that the process is also language based, but I think it's not, especially when doing hard stuff.

I know for certain in my case it isn't -- when tracking a hard problem for a junior after 2 hours of pair programming the other week, I had to tell him to commit everything and just let me do some deep thinking/debugging and I solved the problem myself. Sure I explained my process to him in language the best I could, but it's clear it was not language, it was not liniar, I did not think it step by step.

I wish I could explain it, but when figuring out a hard problem, for me it takes some time to take it all in, get used to the moving parts, play with them. I'm sure there are actual neurons/synapses formed then, actual new wires sprawling about in the brain, that's why it takes time. I think the solution is a hardware one, not a software one.

That's why we can sleep on it and get better the next day and that's why we feel the problem. There are actual multiple paralel "threads" of thinking going at the same time in our heads and we can FEEL the solution as almost there.

I think it simply is that hard problems can occur in a combination of code, state, models that simply cannot be solved incrementally and big jumps are necessary.

I'm not saying the problem cannot be solved incrementally, but it's possible that by going in small steps, you either reach the solution or a blocker that requires a big jump.

Re:GPT-5 for Developers by beoberha on Thursday August 07, 2025 @02:09PM (#44828193)
A series of small manageable chunks becomes a long running task :)

If LLMs are going to act as agents, they need to maintain context across these chunks.

Re:GPT-5 for Developers by vaenaes on Thursday August 07, 2025 @02:34PM (#44828563)
You're holding it wrong
Re:GPT-5 for Developers by joshmlewis on Thursday August 07, 2025 @05:46PM (#44830772)
I've been testing it against Opus 4.1 the last few hours and it has done better and solved problems Claude kept failing at. I would say it's definitely better, at least so far.
Re:GPT-5 for Developers by cyanydeez on Thursday August 07, 2025 @06:09PM (#44830986)
real context is a graph of objectives and results.

The power of these models has peaked and simply arn't going to manage the type of awareness being promised.

Re:GPT-5 for Developers by 1659447091 on Thursday August 07, 2025 @06:45PM (#44831325)
> Producing a very complex, context-exceeding objective is a daily (maybe hourly) ocurrence for me. All I care about is how these systems manage context and stay on track over extended periods of time.

For whatever reason Github's Copilot is treated like the redheaded stepchild of coding assistants. Even through there are Anthropic, OpenAI, and Google models to choose from. And there is a "spaces"[0] website feature that may be close to what you are looking for.

I got better results for testing some larger task using that than I did through the IDE version. But have not used it much. Maybe others have more experience with it. Trying to gather all the context and then review the results was taking longer than doing it myself; having the context gathered already or building it up over time is probably where its value is.

[0] https://docs.github.com/en/copilot/concepts/spaces

Re:GPT-5 for Developers by ilaksh on Thursday August 07, 2025 @06:58PM (#44831423)
The pricing is dramatically better than Opus for gpt-5 since that is now comparable to Gemini 2.5 Pro.
Re:GPT-5 for Developers by user3939382 on Thursday August 07, 2025 @11:27PM (#44833076)
Sorry if this is repetitive but you have to break the problem down just like any complex computing task. The difference is how. You have to break the problems into context windows that you anticipate being able to sow together later. It’s not the same way you would break down a source code authoring task in its absence but the theory is the same.
Re:GPT-5 for Developers by altitudinous on Friday August 08, 2025 @12:55AM (#44833569)
Indeed context awareness is the big difference here, GPT5 is a vast improvement. It doesn't lose track (as easily)
Re:GPT-5 for Developers by greymalik on Friday August 08, 2025 @10:36AM (#44837505)
> it's not clear there's a substantial difference in software development expertise

But GPT-5 is substantially cheaper[0].

[0] https://simonwillison.net/2025/Aug/7/gpt-5/#pricing-is-aggre...

Re:GPT-5 for Developers by andhuman on Friday August 08, 2025 @10:56AM (#44837775)
Is this because it’s now a Moe? They now match price with Gemini 2.5 Pro, which is also a moe.