Serious question: How do you assess this? It seems to me that it would take very significant time to establish that conclusion.
I find it hard to believe they would spend the effort and money making an AI do tasks like that, but are ALSO willing to spend the effort and time to properly validate the changes. I’m sure many do… but probably not a large percentage.
I mainly look at:
- The overall architecture to make sure it's in line with what I want
- The points where it interacts with certain other systems I'm concerned about
And I test the feature myself in addition to the test cases. Sol has proven to be outstanding at good test cases and having a coherent view on overall architecture integration. I basically never think to myself "this entire implementation is slop, I need to rip it out and refactor it" anymore.
I suspect the model's capacity to translate requirements into contextually appropriate test cases, and insistence on thoroughly covering functionality in tests, is a large part of its strength.
Did you feel the same way about 5.5? The same general sentiment keeps being expressed with every model release. The prior generation is immediately cast aside with a vague “well yes, actually we didn’t mean it last time” attitude.
I think I even heard Theo the T3 guy suggesting that his work comes to halt if he can’t use the latest generation model, despite heaping significant praise on each generation previous and suggesting it does all of his programming tasks.
The boundless energy with which each model generation is said to be revolutionary and leaving everyone behind is tiresome even when incremental progress is real and measurable.
5.5 on Extra High I found to be a huge leap in what it could tackle and how effectively but only up to a point. Still had common failure modes here and there but a big improvement
When I tried 5.6 I wasn't expecting more than an incremental upgrade, but it blew me away with how "smart" it seemed, how it completely sidestepped failure modes I'd been used to dealing with in 5.5, its success rate on the first try for wide-sweeping architectural tasks, and how I've basically never seen a bad hallucination make it through to the final output. It's the first time I've felt like if I left it with a task on the highest setting and it had any way to verify its output, and told it to keep going, it could solve almost anything. This could all be incremental upgrades in many different areas of the model+harness, but the cumulative effect feels like crossing that "escape velocity" threshold. The point where there are no more rough edges and it "just works".
I can't speak to others, but my feelings on all of these have been consistent since I tried them in the first place. I've also heard model providers draw people in at each new release with the model at full performance and then degrade it over time hoping users won't notice, which has seemed plausible anecdotally but I have no hard proof.
But truly I'm mad at the vscode harness. I used to be able to follow the 'thought' of the AI model and steer them when I saw a mistake, it's now way harder to do as multiple research tasks are done in parallel.
Preposterous. I can tell 5.6 Sol to do something like "optimize this entire subsystem to have no ongoing memory allocations, fix these 5 bugs, implement these 2 features" and go work on other code or do something else entirely, and it does it all flawlessly. All things I could have done myself but nowhere near as fast and right the first time with such thorough test cases. We've reached the point where the AI is usually faster and always less effort overall compared to the old days, even if you have to send a few followup prompts.