I'd think Win32 development would be something AIs are very strong at because it's so old, so well documented, and there's a ton of code out there for it to read. Yet it still struggles with the differences between Windows messages, control notification messages, and command messages.
It's also another in my growing list of data points towards my opinion that if an author posts meme pictures in their article, it's probably not an article I'm interested in reading.
The tools really do shine where they're good though. They're amazing. But the moment you try to do the more "serious" work with them, it falls apart rapidly.
I say this as someone that uses the tools every day. The only explanation that makes sense to me is that the "you don't get it, they're amazing at everything" people just aren't working on anything even remotely complicated. Or it's confirmation bias that they're only remembering the good results - as we saw with last week's study on the impact of these tools on open source development (perceived productivity was up, real productivity was down). Until we start seeing examples to the contrary, IMO it's not worth thinking that much about. Use them at what they're good at, don't use them for other tasks.
LLMs don't have to be "all or nothing". They absolutely are not good at everything, but that doesn't mean they aren't good at anything.
But I think we should expect the scope of LLM work to improve rapidly in the next few years.
https://metr.org/blog/2025-03-19-measuring-ai-ability-to-com...
Sorry, but this is just not true.
I'm using agents with a totally idiosyncratic code base of Haskell + Bazel + Flutter. It's a stack that is so quirky and niche that even Google hasn't been able to make it work well despite all their developer talent and years of SWEs pushing for things like Haskell support internally.
With agents I'm easily 100x more productive than I would be otherwise.
I'm just starting on a C++ project, but I've already done at least 2 weeks worth of work in under a day.
If you honestly believe that "agents" are making you better than Goole SWEs then you severely need to take a step back and reevaluate, because you are wrong.
If you’re really that more productive, why don’t you quit your job and vibecode 10 ios apps (in your case that would be 50 to 100 proportionally)
This doesn't mean that it's not useful, or that you shouldn't be happy with what the LLM built. I also had Claude Code build me a web app for my own personal use in Rust this week. It's very useful to me. But it is 100% of POC/MVP quality, and always will be, because the code that it created is abjectly awful and I would never be able to scale it into a real world service without rewriting 50+% of it.
It’s nice because 3/4 of those are well-known but not “default” industry choices and it still handles them very well.
So there’s a Laravel CRM builder called Filament which is really fun to work in. Claude does a great job with that. It’s a tremendous amount of boilerplate with clear documentation, so it makes sense that Claude would do well.
The thing I appreciate though is that CC as an agent is able to do a lot in one go.
I’ve also hooked CC up to a read-only API for a client, and I need to consume all of the data on that API for the transition to a Filament app. Claude is currently determining the schema, replicating it in Laravel, and doing a full pull of API records into Laravel models, all on its own. It’s been running for 10 minutes with no interruption and I expect will perform flawlessly at that.
I invest a lot of energy in prompt preparation. My prompts are usually about 200 words for a feature, and I’ll go back and forth with an LLM to make sure it thinks it’s clear enough.
In my experience, LLMs are great at small tasks (bash or python scripts); good at simple CRUD stuff (js, ts, html, css, python); good at prototyping; good at documentation; okay at writing unit tests; okay at adding simple features in more complex databases;
Anything more complex and I find it pretty much unusable, even with Claude 4. More complex C++ codebases; more niche libraries; ML, CV, more mathsy domains that require reasoning.
Not to dog the author too hard, but a look at their Github profile says a lot about the projects they've worked on and what kind of dev they are. Not much there in terms of projects or code output, but they do have 15k followers on Twitter, where they post frequently about LLMs to their audience.
They aren't talking about the tasks and the domains they're using because that's incidental; what they really want to do is just talk about LLMs to their online audience, not ship code.
People have such widely varying experiences and I’m wondering why.