Getting things done require a lot of booksmarts, but also a lot of "street smarts" - knowing when to answer quickly, when to double back, etc
TL;DR - If you're writing more than one failing test at a time, you are not doing Test-Driven Development.
In general, benchmarks seem to very misleading in my experience, and I still prefer sonnet 3.5 for _nearly_ every use case- except massive text tasks, which I use gemini 2.0 pro with the 2M token context window.
Claude is pretty consistently the chat I go back to where the responses subjectively seem better to me, regardless of where the model actually lands in benchmarks.
There absolutely is, even when it isn't intended.
The difference between what the model is fitting to and reality it is used on is essentially every problem in AI, from paperclipping to hallucination, from unlawful output to simple classification errors.
(Ok, not every problem, there's also sample efficiency, and…)
I’d highly recommend anyone check out Windsurf’s Cascade feature for agentic-like code writing and exploration. It helped save me many hours in understanding new codebases and tracing data flows.
For example, DeepSeek’s models almost always smash together C++ headers and code files even with Qt, which is an absolutely egregious error due to the meta-object compiler preprocessor step. The MOC has been around for at least 15 years and is all over the training data so there’s no excuse.
That being said, when cost isn't a factor Claude remains my winner for coding.
I especially want to publish and send you the code for that inspector class and selector GUI that dumps the component hierarchy/state, QML source, and screenshot for use with Claude. Sadly I (and Claude) took some dumb shortcuts while implementing the inspector class that both couples it to proprietary code I can’t share and hardcodes some project specific bits, so it’s going to take me a bit of time to extricate the core logic.
I haven’t tried it with 3.7 but based on my tree-sitter QSyntaxHighlighter and Markdown QAbstactListModel tests so far, it is significantly better and I suspect the work Anthropic has done to train it for computer use will reap huge rewards for this use case. I’m still experimenting with the nitty gritty details but I think it will also be a game changer for testing in general, because combining computer use, gammaray-like dumps, and the Spix e2e testing API completes the full circle on app context.
From time to time I'm fooling with the idea of open sourcing the core block editor but I don't really get into it since 1. I'm a little embarrassed by the current unmodularization of the code and want to refactor it all. 2. I want to still find a way to monetize my open source projects (so maybe AGPL with commercial license?)
Dude, that inspector looks so cool. Can't wait to try it. Do you think it can also show how much memory each QML component is taking?
I'm hyped as well about Claude 3.7, haven't had the time to play with it on my Qt C++ projects yet but will do it soon.
About to push a new release that makes it the default.
It costs money but if you're writing code to make money, it's totally worth it.
This is good news. OpenAI seems to be aiming towards "the smartest model," but in practice, LLMs are used primarily as learning aids, data transformers, and code writers.
Balancing "intelligence" with "get shit done" seems to be the sweet spot, and afaict one of the reasons the current crop of developer tools (Cursor, Windsurf, etc.) prefer Claude 3.5 Sonnet over 4o.