News for hackers. Stuff that matters.
Re:The Claude Delusion by kypro on Saturday November 14, 2026 @10:15AM (#50497687)
I want to preface what I'm about to say with the fact that I believe super intelligence is (by some reasonable definition) more likely than not to arrive some time next year.

However, today's models have extremely jagged intelligence even within fields they're extremely capable in.

Like most people here, I use Claude and other models daily and I used them primarily in the context of building a very complex software project. Claude is brilliant at writing code at, in very clean and well defined codebases, where its being asked to build something that fits well within the preexisting patterns.

Someone non-technical who does a lot of vibe coding recently asked why I couldn't just tell Claude to go build some feature into the project and call it a day? And it's genuinely hard to explain to people who are not both experts broadly in the field and experts in the specifics of the problem why this such a bad idea. And annoyingly it's not that Claude won't produce something that works, it will produce something that makes several extremely bad assumptions and defensively codes in those assumptions. And even when you explain this, the question then changes to, "yeah, but you could just ask Claude to fix that".

My default way of working today is to start 90% of my projects by simply asking in a few paragraphs what I want, seeing what Claude builds then spending half a day working through it, explaining all the wrong assumptions it's made, then reaching a clean and maintainable solution at the end.

However, I have been on the other side of this and I've felt magic of Claude when I've got it to write code I don't really understand. On the surface it seems to be able to do everything I ask stupidly fast. It's better than any human you could ever work with. But who knows what it's doing under the hood... My guess is that you simply can't keep building like that for long because while it will do everything you ask, the foundations of what it's building on are basically non-existent – and anyone whose worked on a serious software project before knows the foundations of any software are by far the most important things in the long-term robustness and maintainability of the project.

I guess what I'm saying is that many people who are non-experts in the field or non-exports in the specific problem within the fields they're asking AI to do things are largely delusional about what AI can and can't do. It absolutely can create a simple CRUD app, but it will not build a complicated distributed system and process millions of requests a day with five 9s of uptime.

That's not to say it will never happen. I think it will, but there's some things that need to be figured out first.

I was working with someone earlier this week who was telling me about how he convinced GPT-6.1 his name was an an XSS attack string. I don't think you could convince 1 in a million humans that your legal name is genuinely <script>alert("hello!");</script> and yet these models are supposed to be experts in domains like cyber security and super intelligent? It's honestly staggering how stupid they can be whilst also being so competent at a surface level.

There are no comments on this comment yet.