And I completely agree about performance optimization btw, though I don't think I would say LLMs are better than humans at security yet because of how poor their epistemics/short their context is.
They're both highly verifiable domains that you could easily train to be superhuman because they're also easy to generate and verify against synthetic tasks. And they probably bring a lot more knowledge about software's internals in-distribution while doing it, so I don't expect to ever be able to out-optimize frontier AI at squeezing performance out of hardware or into software.
But I would be cautious to deferring to their judgment in long-horizon architecture or the more procedural, threat-modelly/incentives side of security, and for other large scale judgment calls because they simply have no experience with them.
RL is turning them into Idiot Savants so jagged in understanding and performance that they defy human expectations for what kind of knowledge and behavior correlate (causally). A human engineer superhuman at hill-climbing low level performance would be at least 95%ile at at general architecture, since they're the same problem just at larger and slower scale.
There's no good hyperbaric benchmaxxing chamber devised for tasks at this level of scope/ambiguity yet. they fall back to platitudes and best practices (the "Reddit"/HN part of the weights). In my opinion, they are just kinda "ok" at architecture. Once you're in the pocket of what they've spent more time learning in parallel than any (or all) humans could, they just let their freak flag fly without hedging or asking about it.
To demonstrate: I had Claude earlier this morning spending 3+hrs pruning a pure x86 ASM implementation of symmetric DTLS/symmetric UDP, down to a 3.8KB binary, completely from scratch. It flew off and ripped. No human alive would or will ever be able do this kind of task as quickly - expand those "66 steps" and see for yourself.
But! Any human capable of doing this task would know to add support for certificates before spending an ungodly amount of effort optimizing a security protocol implementation that needs them but doesn't have them yet! Claude has just never done anything greater in scope than 1M tokens so: https://claude.ai/share/90de9350-81f9-46ea-a628-be625e8b9a7a
> and I found in review that Claude would leave the debugging tooling around unflagged
Why is Claude to blame for that particular decision, if you created it? That's what this is about - you need to constrain the design space to take responsibility. In this case, understand the debug tooling and what it should be doing. Negative prompting is rarely a good solution.
These foibles come and go: early Claude Code refused to let programs crash, mangled all the comments in your codebase, and then one week you find that it insists on ending all functions with `Of` as in `layoutOf()`, `viewOf()`. Finding and mitigating these annoying habits becomes part of the job. Code review becomes articulating with this entity why certain patterns in the code are silly and should be changed.
The author mentions performance and security issues as the domain of “the architect.” For performance, I’ve found that Claude Code has more or less solved this domain, and can profile and optimize code across a variety of stacks and languages using sophisticated, ad-hoc workflows. For security, it’s just an ever-ending nightmare where random habits and sloppy implementation create random vulernabilities, compounded by the fact that Anthropic has added “safeguard filters” whenever you naively say you’re looking for them. Nevertheless, I’m not sure the architecting role as they describe won’t be automated in the near future.