Re:How the Claude Code team thinks about verification
by TobTobXX on Tuesday November 17, 2026 @05:53AM
(#50529823)
Apparently, the remaining 10% too, but not attributed correctly, so they are careful of claiming it as Claude-written. (Can't remember where I've read that.)
Re:How the Claude Code team thinks about verification
by thatsabadlook on Tuesday November 17, 2026 @06:16AM
(#50529960)
You should fire yourself and have your boss go all in on anthropic duh. If you are the boss then you either shouldn't have gone down this path, accept your code won't be verified, or shut your doors.
Re:How the Claude Code team thinks about verification
by 4lb3rn on Tuesday November 17, 2026 @06:24AM
(#50530019)
Exactly this. I build large systems like erp, ipaas just from a Claude team with two premium licences. everything builds, verifies, deploys itself. I can track everything and implement features in a days. security and big fixing is background work. I would love to know what 2000$ worth of tokens do in your business.
Re:How the Claude Code team thinks about verification
by bcrosby95 on Tuesday November 17, 2026 @07:21AM
(#50530438)
I don't know if this is sarcasm, but token maxing is largely over, at least anecdotally, according to my friends across several large companies: surpassing $1k-$2k/mo is roughly when they start asking their developers "what the fuck you doing".
Re:How the Claude Code team thinks about verification
by duskdozer on Tuesday November 17, 2026 @06:56AM
(#50530250)
Why aren't you spending $12k on tokens? Or better, $200k? Coding is solved, you can now infinite money glitch; just give your current money, which will be useless in the Agentic Future anyway, to Anthropic now.
Re:How the Claude Code team thinks about verification
by xnorswap on Tuesday November 17, 2026 @07:00AM
(#50530282)
( NB: This is a joke, not a serious suggestion! )
If you want the fraudulent route:
Spin up a sub company, invent some paper employees, spend $2k on "Teams" seats ( ~40 seats? ) for them.
In my experience, each seat can run up 5-10 parallel agents within the monthly usage.
That's 200-400 agents for similar money. It requires some fraud or willingness to hire cheap labour to fill the seats.
On a more serious note, I genuinely feel bad for any company too large to take advantage of the teams plan, I've never seen a more generous subscription offering in my life when compared to API pricing and it still shocks me that enterprises are forced onto API pricing, but I guess it's heavily subsidising SMEs and therefore possibly helping the economy by funding innovation through more nimble businesses.
Re:How the Claude Code team thinks about verification
by simianwords on Tuesday November 17, 2026 @06:57AM
(#50530258)
HN comments love to use the curmudgeonly tone but did you not stop to think that they are going as far as possible to let model do what it can and do everything else?
Did you forget that they literally publish benchmarks that show how much of the task can be done independently?
Re:How the Claude Code team thinks about verification
by mohsen1 on Tuesday November 17, 2026 @06:20AM
(#50529990)
Verification is the bottleneck and that's why we're building Proofline.sh
Starting with infra evidence and soon expanding to application verification loops. This is a very exciting place to build and we're seeing great results once the verification loop is closed for the teams
Re:How the Claude Code team thinks about verification
by ahsillyme on Tuesday November 17, 2026 @06:46AM
(#50530166)
Issues are automatically closed after being inactive for too long. It's also true that many issues that are filed there have nothing to do with Claude Code proper. To conclude, these numbers (either many open, or many closed, or any relationship between them) a-priori mean nothing either way.
Re:How the Claude Code team thinks about verification
by OtherShrezzing on Tuesday November 17, 2026 @06:35AM
(#50530087)
This article feels like the result of having unlimited tokens at your disposal. It’s got some very expensive practices, where near-free practices would produce the same or better results.
There’s a long section in the middle where the article describes per-change verification via skill files and “ask Claude to use the chrome-for-Claude tool” when checking if changes have introduced regressions.
But there’s already tools which do this without spending tokens tokens. Playwright will achieve equal quality but deterministic results across the same verification attempts. It is write-once-use-often, so has a marginal cost near zero.
Re:How the Claude Code team thinks about verification
by ben_w on Tuesday November 17, 2026 @07:04AM
(#50530306)
Quite possibly. But, a thought I had some years back:
In the future when AI is able to do everything, there will not be one single super-AI that does it all, but a whole ecosystem of different AI. This is the same as how in the biological world, while there are various degrees of generalists, there's no single cell, plant, or animal which does absolutely everything. If our body needs more blood, it makes more blood cells, it doesn't repurpose intestine or skin for the task.
Similar argument in economics, comparative advantage: an AI which can do everything is best used for the things which only it and no other AI can do, and leaving to other (smaller, cheaper) AI relatively simple tasks like face recognition.
Even where AI can do this, that doesn't mean the best solution is AI rather than a traditional API.
Re:How the Claude Code team thinks about verification
by ben_w on Tuesday November 17, 2026 @07:23AM
(#50530459)
Hardly.
First, last I checked AlphaZero is a much better chess player than any LLM, and by a large margin.
Second, this isn't even about finding the best model for small tasks, it's about best use of finite resources, compute or energy, which means that it still holds even if a future AI (call it Alice) did everything better than AlphaZero does chess, it would still make sense for Alice to offload chess to a merely-good model like AlphaZero so that Alice could spend more time doing the things that nothing else besides Alice could do:
Re:How the Claude Code team thinks about verification
by antonvs on Wednesday November 18, 2026 @03:21AM
(#50543377)
That’s not what the Bitter Lesson addresses. It looks at where advances in AI can come from and how different approaches, scaling vs. applying human knowledge to develop fancier models, compare in advancing AI. It doesn’t deal with comparative operating costs of different sizes of model.
Small, specialized models can still benefit from the Bitter Lesson in that they can take advantage of ever-cheaper computation. They don’t contradict the lesson.
Re:How the Claude Code team thinks about verification
by slowhorse on Tuesday November 17, 2026 @07:12AM
(#50530360)
So the future is about replacing deterministic procedures with stochastic procedures?
On some level you are right, LLMs and the agents built with the various models and architectures will get cheaper. But there are some significant drawbacks by introducing this risk, regardless of cost. The future will need the capacity to understand this too.
Re:How the Claude Code team thinks about verification
by mleo on Tuesday November 17, 2026 @09:37AM
(#50531976)
Just as likely, as models and harnesses get better with memory, they will recognize the process of building and rebuilding the same tool to complete a task can be optimized by keeping the tool around and using it to deterministically complete the task instead.
This is the process of increasing efficiency today, automate manual processes to reduce human effort. Why wouldn’t models get better at this as well?
Re:How the Claude Code team thinks about verification
by adamddev1 on Tuesday November 17, 2026 @07:00AM
(#50530281)
I do understand that it is extreme, but I think the principled stand makes my code and my skills better in the long run. I'm banking on building less, but building better. Otherwise where do you draw the line in losing quality and understanding? And for what I'm working on, typing is rarely the bottleneck anyways.
I still use LLMs (cautiously) for searching and troubleshooting API things.
Re:How the Claude Code team thinks about verification
by catchnear4321 on Tuesday November 17, 2026 @07:44AM
(#50530651)
Letting go of the typing isn’t a light decision. And it is harder to get good output when it’s also simply a lot of output.
I’m banking on that being an evolving, but necessary skillset. For a while at least. And I’m still gaining experience in the process.
Plenty of exposure, plenty of prior experience, it’s a weird mix of management-ish context handling and sme mentorship while also providing the research benefits you mentioned.
A key pillar was introducing things like static analysis to provide cheap deterministic checks as the first goal and only then moving to other forms of verification.
Re:How the Claude Code team thinks about verification
by simianwords on Tuesday November 17, 2026 @06:53AM
(#50530230)
You are not agreeing with the OP. You are just saying that there are many levels to testing and each give their own swiss cheese level of verification.
Sure you can have linting, compilation, unit tests, e2e tests. Do you really think Anthropic doesn't do this?
Re:How the Claude Code team thinks about verification
by xnorswap on Tuesday November 17, 2026 @07:06AM
(#50530318)
I was agreeing that this page as presented felt like it was from the POV of people will unlimited tokens to burn.
In particular constructing ( and often deconstructing on change ) differential tests is very expensive relative to cheaper things to set up.
I felt this article's argument was poorly presented because of this seeming contradiction. The video I linked was much clearer about what works and what doesn't when scaling to inhuman levels of PR merges.
Re:How the Claude Code team thinks about verification
by simianwords on Tuesday November 17, 2026 @07:18AM
(#50530416)
I don't agree with "very expensive" claim. My friend works in a f500. They do releases like 2-3 times a week. Assuming each release uses $500 of tokens (higher estimate) just for verification, that's like $6000 a month and that's just a small percent over the tokens used for coding. What's the big deal? Their team brings in way more money than that - its a drop in the ocean.
I also know of companies that run very expensive test suites using spot instances in AWS that cost around $100 each time. No one complains about it.
I think all the posts complaining about expenses forget to compare the orders of magnitudes.
Re:How the Claude Code team thinks about verification
by xnorswap on Tuesday November 17, 2026 @07:49AM
(#50530702)
It's not the per-run costs I worry about, it's the costs of churn.
As soon as Oracle Tests fail, the LLM has three options:
1. Rewrite the test - This undoes the benefit of the test in the first place
2. Implement workarounds and quirks to match the oracle, often at the expense of recreating buggy behaviour
3. Implement both corrected behaviour and quirks-mode behaviour to meet the oracle, raise further jobs to formalise the switch over and change of behaviour.
( And this "change of behaviour" might be a few pixels out of place! Or it could be a much more significant change. When trying to meet a goal, LLMs will often treat these with the same zeal. )
If you're working interactively, you can catch it and steer it to what's best for the feature. If you're fully automating development, then you're at the mercy of whatever the model does at that moment.
Oracle-based testing has its place, but I don't think it's good for large scale automated testing.
Also, you're talking about $500/release 2-3 times a week. That's fine, that's less than the cost of an employee.
Anthropic and others are talking about how to scale to merge 2000-3000 PRs per week.
At that scale, that cost per merge is $6M/month, surely a cost you'd want to minimise if you can cut them by half with better type-checking, static analysis, and other cheap deterministic tests, even if that's still a small cost of the whole project.
If nothing else, it's pure waste. Our planet is (mostly metaphorically) burning, and I believe that more efficient IT processes and development practices would go a long way to helping mitigate some of that.
If one’s original argument doesn’t have enough weight, they just add other weak ones to justify it.
Anyway, responding to the strongest version of it: the economics of any new system always looks bad before it becomes popular. I heard people say the same thing about AI driven coding and now there’s absolutely no going back. If you were a software engineer living in 1990s and you heard someone say that Uber, a ride sharing company, spends 100s of millions per year on just microservices and db infra, you’d be skeptical again.
Usually the costs justify themselves because of the productivity it brings. People spend 10x on cloud for this reason.
If you think I’m wrong, let’s make a bet of sorts. I’m highly confident that AI driven testing will be another normal and extensively used tool just like how CI/CD as a concept is today
Re:How the Claude Code team thinks about verification
by xnorswap on Tuesday November 17, 2026 @08:12AM
(#50530901)
Slopping out 2k+ PRs/mo while also claiming to reduce defect rates.
That seems like something to pay attention to, even if you prefer to treat it as a cautionary tale with your popcorn at the ready for when you expect it all comes crashing down.
Re:How the Claude Code team thinks about verification
by rolosa on Tuesday November 17, 2026 @08:49AM
(#50531344)
At 2000 PR/mo, assuming a "standard" 8 hour per day, 5 days a week workweek with no time factored for lunch, meetings, breaks... literally all they do for 8 hours a day, 5 days a week, is submitting PRs. That works out to roughly 5 minutes to slop, review, and document a single PR, approx 92 PRs per work day. That tells me they aren't doing shit.
Re:How the Claude Code team thinks about verification
by xnorswap on Tuesday November 17, 2026 @09:09AM
(#50531596)
I suspect that an ever increasing number of tickets being raised are being raised by agents themselves, nitpicking and being unable to deal with the idea of implementations having trade-offs.
It doesn't take 5 minutes to prompt claude, "Please pick up Ticket #1234567".
With modern models, you rarely need much more than that. Perhaps checking in an hour later with a quick prompt, "Progress report?" if it's still working on it.
Claude having written the ticket in the first place, defined acceptance criteria. Perhaps a human has skimmed it for a go/no-go decision, perhaps not.
The point is that at that scale you can't humanly read the code, so you need to define verification strategies that work for the machine.
Precisely so that you're not just running on pure vibes.
Re:How the Claude Code team thinks about verification
by xnorswap on Tuesday November 17, 2026 @09:59AM
(#50532270)
I don't think it is, but I think it's something worth paying attention to.
With my skeptic hat on, it'll be interesting to see what happens when you look at their internal speed, in PRs/month vs the outward velocity in features and improvements actually shipped.
With my booster hat, it's something to learn from so I can better scale from juggling hand-reviewed claude sessions to a future it might be demanded I join at some point.
We're in a weird space in development, where the initial mechanical work that used to take a week now takes an hour at most, but what remains is a lot of manual verification and developer-testing.
If we can learn from what these mega-swarms are doing to reduce the need for manual developer testing ( because at that speed it's clear there must be close to none ), then that is a useful lesson that might reduce our work per task too and let us do more.
And yes, some of that may be stuff we've known for years, such as functional CI/CD, better linting, better automated e2e testing with playwright, etc.
But actually convincing management to invest in that kind of thing isn't always easy, especially if it risks instability of a process that from their perspective has been working fine for decades, even if that "process" is someone logging onto a server to manually fire off a script that was last touched 15 years ago.
So having talks such as this to point to is useful. You can show management an expert on a stage, and in some companies an expert on a stage can be weirdly convincing to management in a way that their own employees sometimes are not.
Re:How the Claude Code team thinks about verification
by skydhash on Tuesday November 17, 2026 @11:01AM
(#50533275)
> We're in a weird space in development, where the initial mechanical work that used to take a week now takes an hour at most, but what remains is a lot of manual verification and developer-testing.
I don't think the low speed of development was an actual bottleneck. Coding in the real world is actually a mix of mechanical typing and information gathering. The typing part is not that relevant as it's actually not much time to type in 500 lines of code (if you touch type) assuming that's the median size of PR. It is the gathering of information that is IMO crucial for the health of the software.
It's like robots in a chemical lab. You can probably build some mechanical apparatus that can mix whatever components you want. But you still need a scientist to design experiments and to check the result. And while you can help speed up some of the process, what's valuable is the knowledge that the scientist extract, not the amount of experiments being done.
This scale of work is only possible because there's no information gathering going on. Instead it's just busy work with wishes and prayers hoping to get a good result.
Re:How the Claude Code team thinks about verification
by fg137 on Tuesday November 17, 2026 @09:48AM
(#50532133)
While I have seen numerous tickets addressed by Claude/Codex in one shot with decent solutions, about half of their first implementation is completely wrong in the direction, is way over engineered/significantly expands the scope, or leads to very confusing or even incorrect user-facing behavior.
I can't see how I would be able to keep customers happy without actually reading the code and understand what it is doing.
Re:How the Claude Code team thinks about verification
by rolosa on Tuesday November 17, 2026 @12:11PM
(#50534616)
The argument that engineers like the one in question are needed because of their judgment, guidance, experience, etc. But on the other hand if their job is reduced down to "@grok review and fix issue #42069", every 5 minutes of their workday, then what value do they actually add? Why not just remove them from the equation? I'm saying that a developer who brags about doing 2000+ prs/day is effectively dead weight and brings nothing to the table.
Re:How the Claude Code team thinks about verification
by senordevnyc on Tuesday November 17, 2026 @08:29PM
(#50540719)
Well, she’s using the new tools available to us to pioneer these kinds of scalable software factories and using that process to build a very successful product.
It sounds like you’re doing things basically the way they’ve always been done.
This is so far from engaging with how Lauren actually works that’s it’s barely worth even responding to. They’re describing a standard software engineer workflow as it existed decades ago. Lauren is playing on a totally different level, more successfully I suspect (though who knows because while she’s public, this critic is anonymous).
Re:How the Claude Code team thinks about verification
by Alacart on Tuesday November 17, 2026 @10:13AM
(#50532477)
While I suspect your point about the value/quality of the work at 2000 PRs/month is directionally correct, isn’t their entire take that they’re no longer micromanaging the PRs at all? So talking about how much time they commit to each one doesn’t make much sense. It’s a bit like looking at a modern car and saying “haha how will they even get the horses to pull this?” or maybe “how do these developers know the compiler isn’t using GOTO statements underneath?”.
Now, are those PRs doing useless work? Are they introducing unnecessary risks or trade offs? Are they building towards something impossible to understand or that will fall apart under its own sloppy weight?
Those seem more like the right questions to me, in the long run, and I suspect that’s what you really have in mind. Because if the answers are no, then why wouldn’t you be doing as many PRs as possible?
Re:How the Claude Code team thinks about verification
by JambalayaJimbo on Tuesday November 17, 2026 @11:06AM
(#50533380)
A PR to me represents a specific boundary of human ... review. It's a unit of work that is typically reviewed by at least 2 team members. If I'm working solo on some project, I don't use PRs.
So to me "not micromanaging PRs" doesn't make any sense at all. Bragging about how many PRs you can get merged is like bragging about how many WPM you can get when prompting an AI instead of manually typing.
Re:How the Claude Code team thinks about verification
by ffsm8 on Tuesday November 17, 2026 @02:15PM
(#50536676)
That's been the way I've worked in previous jobs too. Nonetheless some PRs need less review, others more. It's a gradient, and they're merely moving the range further towards "doesn't need in-depth" by introducing other quality gates.
That comes with trade-offs, but some may be willing to accept those. It's ultimately the decision of the developers and enterprise how they want to build their product, can't really say anything else about this
Re:How the Claude Code team thinks about verification
by jimbokun on Tuesday November 17, 2026 @05:15PM
(#50539024)
PRs are part of a software development process meant for collaborating humans. If no humans are reviewing anything, having PRs as part of the process is pretty pointless.
Re:How the Claude Code team thinks about verification
by jwilliams on Tuesday November 17, 2026 @07:03PM
(#50540051)
I mean, I agree conceptually. But the PR is just a useful unit of work.
My PRs get assessed for if a human is needed. I could probably do that at the branch level, but PRs just has all the metadata and CI/CD tooling etc etc.
Re:How the Claude Code team thinks about verification
by dmoy on Tuesday November 17, 2026 @05:35PM
(#50539258)
> Or that human review is mandatory?
Depending on the domain, human review can be mandatory. (i.e. if you're a public company and the thing you're touching can materially affect the company finances)
Re:How the Claude Code team thinks about verification
by fg137 on Tuesday November 17, 2026 @08:03AM
(#50530817)
The talk makes that sound something new, but every team I know has already adopted that before AI. I really don't get what value that talk provides -- I learned nothing after watching it.
Re:How the Claude Code team thinks about verification
by xnorswap on Tuesday November 17, 2026 @08:09AM
(#50530868)
You are fortunate to have always worked in highly effective teams and organisations.
This reminds me of when "The Joel Test" was first published and it was met by people unbelieving there was a world out there who weren't even using source control at the time, let alone a bug-tracker.
Re:How the Claude Code team thinks about verification
by JambalayaJimbo on Tuesday November 17, 2026 @11:00AM
(#50533264)
I have a friend who worked for a billion dollar company (a household name here in Canada) who conferred that they weren't using source control. As late as 2019.
Re:How the Claude Code team thinks about verification
by vinnymac on Tuesday November 17, 2026 @02:03PM
(#50536492)
Reminds me of a DevOps conference I attended in 2019 where everyone in the room hadn't heard of a feature flag before, and wanted me to share my experiences with them so they could better understand how to use them in their enterprise software. Meanwhile I had been using them since I was an intern, and was confused why all these graybeards were listening to me talk about anything from a db table to a full fledged company like LaunchDarkly, Optimizely, etc.
Re:How the Claude Code team thinks about verification
by jamesfinlayson on Tuesday November 17, 2026 @11:28PM
(#50541883)
I hope their beards really were grey - I work with a guy who worked somewhere in the mid 1990s and there were two servers (mainframes) - a development server and the production server. I could believe feature flags weren't a thing at that company, but we've also come a long way.
Re:How the Claude Code team thinks about verification
by SatvikBeri on Tuesday November 17, 2026 @03:40PM
(#50537860)
For some contrast, my first job where the team used source control was in 2015 – 15 years after the Joel test came out. That was after switching jobs in manufacturing or finance to tech.
Re:How the Claude Code team thinks about verification
by stingraycharles on Tuesday November 17, 2026 @06:40PM
(#50539829)
Believe it or not, there still are so many places that I’m working with that have hundreds of thousands of lines of code and no source control. They just pass zip files around and whatnot. There’s a whole parallel universe invisible to the average HN reader.
Re:How the Claude Code team thinks about verification
by verelo on Tuesday November 17, 2026 @10:19AM
(#50532564)
I wish my reaction to seeing this post was "oh wow Anthropic never write stuff like this, I should make time to read it". My gut instinct was to say "this is going to be long, over complicated and not worth my time". I really think this is the influence of the 90's - 00's SEO. Write lots of stuff, hope Google cares...get more business.
Re:How the Claude Code team thinks about verification
by arbll on Tuesday November 17, 2026 @06:50AM
(#50530199)
What makes you think they are not using Playwright ?
For me this article is more aimed at devs that are still stuck treating code generation and verification as two separate tasks. Giving agents the right tools (including playwright) to do both at once makes things more token efficient, not less.
Re:How the Claude Code team thinks about verification
by baxtr on Tuesday November 17, 2026 @06:52AM
(#50530214)
Wait, are you saying a company that makes money from selling tokens generated a content piece recommending people solve their problems by using a lot of tokens?
Re:How the Claude Code team thinks about verification
by simianwords on Tuesday November 17, 2026 @06:52AM
(#50530219)
Its almost become a meme at this point: well actually you can achieve the same using X so why west tokens?!
The reason you should use Claude and not playwright always is because the the invariant you wanna capture is at the natural language level.
Like imagine I wanted to verify that Claude Code works and doesn't break. I can describe the invariant I want in natural language. Like: make sure that subagents don't block the original thread.
Doing this in playwright means that you have already bought into some previously existing architecture which means your tests are now correlated to your code. Having the invariants written in natural language allows you to decouple code vs the invariant.
I expect a lot of greybeard conservative programmers to keep opposing using AI here until eventually, not using AI will be seen as more risky.
Re:How the Claude Code team thinks about verification
by simianwords on Tuesday November 17, 2026 @12:05PM
(#50534507)
If you re architecture threads to processes, you have to rewrite your tests.
That was kind of my point. Natural language almost always survives rearchitectures because your higher level interface to your end customer rarely has churn.
In a few years, agent driven testing will become normal and almost expected in some enterprise companies. Obviously we will still have people like you opposing it on grounds of “but just write tests in playwright”.
I also find the greybeard archetype fascinating. They’ve opposed almost everything new including Cloud, microservices, kubernetes. Now with AI. And here in this thread with AI driven tests. They seem to think programming should be done exactly how it was done when they were in the prime of their career. Anything before that is archaic and anything after is hype.
Re:How the Claude Code team thinks about verification
by simianwords on Tuesday November 17, 2026 @06:06PM
(#50539519)
Maybe. But these new concepts won and they weren’t just hype. They have their place. I’m obviously not saying AI testing has no wrong way of being used.
I’m saying that a default response to challenge AI testing by saying “just write normal tests” is the typical greybeard archetypical response. And they don’t really have a good track record considering they’d by default challenge cloud, kubernetes etc
Re:How the Claude Code team thinks about verification
by saagarjha on Tuesday November 17, 2026 @08:22AM
(#50531012)
Fun fact: Codex computer use is trained to not enter passwords. When it was being designed this was considered to be a very cheap way to ensure the model would not go poking around places it shouldn’t be, instead of constantly trying to check whether the action being performed was authorized or not. People have spent a lot of time marking out their UIs for sensitive actions and it’s worth using it :)
Re:How the Claude Code team thinks about verification
by pu_pe on Tuesday November 17, 2026 @07:12AM
(#50530359)
> It’s got some very expensive practices, where near-free practices would produce the same or better results.
I think tokens are near-free already, even for those of us outside Anthropic.
More importantly, your philosophy comes from a time where thinking had to be front-loaded: you try to predict everything that can go wrong, then design deterministic tests that can catch those things and a decision tree to figure out what to do in each case. This approach was the only viable way until now, but maybe that will change. Having the equivalent of a senior engineer thoroughly review each error with full context of what the application and feature is trying to achieve could perhaps uncover different insights, or cover edge cases more smoothly.
Re:How the Claude Code team thinks about verification
by skydhash on Tuesday November 17, 2026 @07:27AM
(#50530492)
> you try to predict everything that can go wrong, then design deterministic tests that can catch those things and a decision tree to figure out what to do in each case
No real software engineer think that way, as things that can go wrong are near infinite. Instead you design the right thing and write tests that verify that it’s indeed the right thing and add some more that are the wrong thing, especially at the edge of definition. Then you rely on the dev to not make very obvious errors like coding something that is not in the requirements or is completely opposite.
Re:How the Claude Code team thinks about verification
by pu_pe on Tuesday November 17, 2026 @08:35AM
(#50531146)
It isn't the same work, though, so the question is whether the added cost is worth it. Yes it costs more, but the price of tokens has fallen 100x in the past two years and it will continue to drop.
Re:How the Claude Code team thinks about verification
by Grimburger on Tuesday November 17, 2026 @08:28AM
(#50531074)
> tokens are near-free already
How long do you think that will last?
Am over 11 billion tokens this month on a subscription. In API usage its close to $5000 and there's still 13 days left in the month.
Eventually the music in musical chairs stops. Already many anecdotal reports of the always opaque limits being lowered across the board. On a gemini pro account for personal use I can use a weeks worth of Argon in about 20 minutes on agy.
Re:How the Claude Code team thinks about verification
by Grimburger on Tuesday November 17, 2026 @10:02AM
(#50532308)
Undoubtedly you are right, except I see it the complete opposite way, companies providing the best models will probably need even more from those willing to pay in the near future, and likely get it from those wanting to ride ahead of the wave.
Re:How the Claude Code team thinks about verification
by svnt on Tuesday November 17, 2026 @10:48AM
(#50533031)
For the majority of tasks, the best models are no longer required.
Many benchmarks are already saturated today on many tedious tasks still being done by humans across the world. Frontier labs are introducing their own budget options prior to IPO, undercutting their revenue, because otherwise they risk losing seats.
Choosing what to do and how to do it is the actual frontier now and is still not apparently well supported in any model. Current so-called decision models are not decision models, they are available-fact-evaluating models.
Re:How the Claude Code team thinks about verification
by Grimburger on Tuesday November 17, 2026 @04:35PM
(#50538521)
> Many benchmarks are already saturated today on many tedious tasks
Exactly my point, think the easy things won't even need 3rd party apis in the near future.
But frontier models only seem to need more and more VRAM to keep winning the race. Feel free to look up the cost of running Kimi K3 open weights today, probably the best estimate known publicly, it's hardly a casual affair.
Re:How the Claude Code team thinks about verification
by solatic on Wednesday November 18, 2026 @04:41AM
(#50544095)
> How long do you think that will last?
Don't forget that it's a hypothesis, not a foregone conclusion, that this will stop. Part of the question is whether demand will continue to rise. If volume increases sufficiently, you can cut prices and improve profits at the same time, as long as you continue to make a profit at the margin.
Moore's Law kept Intel and competitors pushing R&D budget that pushed perf/watt year after year because the overall computing market just kept growing and growing. The growing demand soaked up everything Intel and co. pushed into the market; the R&D spend helped players retain market share in the face of competition. Why do you think AI is different? R&D into better, faster, cheaper chips brings down the cost of both training and inference over time, and cheaper token cost unlocks additional demand, which justifies more R&D budget, in a virtuous cycle. It continues until there's really no more demand to be unlocked, no more improvements can be made to the chips (i.e. you hit the laws of physics), or there's no more money to be invested (i.e. even capital markets have limits, with only so much cash in circulation, or macro interference like war).
Do we really have any reason to believe that demand will taper off?
Re:How the Claude Code team thinks about verification
by kuschku on Tuesday November 17, 2026 @02:09PM
(#50536587)
Near-free? Even using Claude just for /code-review constantly leads to
> ● Agent "[REDACTED] audit" failed: Agent terminated early due to an API error: You've hit your 5-hour limit · resets 10:50pm (Europe/Berlin) (error type rate_limit, HTTP 429, request id [REDACTED], model sent to the API: claude-opus-5-5)
> ● Agent "Verify [REDACTED] bug" failed: Agent terminated early due to an API error: You've hit your 5-hour limit · resets 10:50pm (Europe/Berlin) (error type rate_limit, HTTP 429, request id [REDACTED], model sent to the API: claude-opus-5-5)
> ● Agent "Verify [REDACTED] bug" failed: Agent terminated early due to an API error: You've hit your 5-hour limit · resets 10:50pm (Europe/Berlin) (error type rate_limit, HTTP 429, request id [REDACTED], model sent to the API: claude-opus-5-5)
> ● Agent "[REDACTED]" failed: Agent terminated early due to an API error: You've hit your 5-hour limit · resets 10:50pm (Europe/Berlin) (error type rate_limit, HTTP 429, request id [REDACTED], model sent to the API: claude-opus-5-5)
> ● Agent "Verify [REDACTED] injection" failed: Agent terminated early due to an API error: You've hit your 5-hour limit · resets 10:50pm (Europe/Berlin) (error type rate_limit, HTTP 429, request id [REDACTED], model sent to the API: claude-opus-5-5)
> ● Agent "Verify [REDACTED] regression" failed: Agent terminated early due to an API error: You've hit your 5-hour limit · resets 10:50pm (Europe/Berlin) (error type rate_limit, HTTP 429, request id [REDACTED], model sent to the API: claude-opus-5-5)
> ● Agent "Verify [REDACTED] race" failed: Agent terminated early due to an API error: You've hit your 5-hour limit · resets 10:50pm (Europe/Berlin) (error type rate_limit, HTTP 429, request id [REDACTED], model sent to the API: claude-opus-5-5)
> ● Agent "[REDACTED] root-cause" failed: Agent terminated early due to an API error: You've hit your 5-hour limit · resets 10:50pm (Europe/Berlin) (error type rate_limit, HTTP 429, request id [REDACTED], model sent to the API: claude-opus-5-5)
> ● Agent "[REDACTED]" failed: Agent terminated early due to an API error: You've hit your 5-hour limit · resets 10:50pm (Europe/Berlin) (error type rate_limit, HTTP 429, request id [REDACTED], model sent to the API: claude-opus-5-5)
Re:How the Claude Code team thinks about verification
by saghm on Tuesday November 17, 2026 @05:44PM
(#50539334)
Even if you aren't paying currency, there's still resource consumption in terms of compute, which in turn requires energy, which is priced in data centers based on usage. You're almost literally taxing people in communities near data centers, I guess because it's cooler to do it this way?
Re:How the Claude Code team thinks about verification
by hypendev on Tuesday November 17, 2026 @07:47AM
(#50530677)
Exactly.
If you're doing this, the agents should create deterministic tests from its process rather than Agent tests or just writes deterministic tests for start. This is a bit of an absurd approach.
The issue I find with these articles is that they are basically the same boat as "Facebook Engineering" articles were 20 years ago.
People will be "oh because Anthropic does it it must be the way", then a bunch of influencers will retweet it as "anthropic kills writing tests", which some CEO's and team leads will parrot and get their team to absorb these practices even tho they are not the best practices.
Some of these things are just reinventing the wheel, like this is a basic script called "run_simulator.sh" and a pre-commit test hook, not a skill and "commit gate" that needs to check session transcript:
>To codify how to start the simulator or run your app, create a skill called verify. Put it in a .claude/skills/ folder in your repo, as close to the relevant code as you can, and add as many as you need. Claude knows to use the verify skill before committing changes.
>We've also created a commit gate that can block a commit if the session did not contain /code-review or /verify. We've set it up with off, observe, and enforce modes.
Re:How the Claude Code team thinks about verification
by urbsgpw on Tuesday November 17, 2026 @08:40AM
(#50531204)
I got this feeling a couple of times this year with their sharing of best practices. They seemed theoretically interesting, but not if you're cost-conscious. It seemed like thinking with an unimited budget. And dont get me wrong, maybe that's the way to go since you assume costs of compute will fall down eventually (and these systems will become more optimized), but it would be completely untenable to run if you don't have a similar budget constraint (or lack thereof).
Re:How the Claude Code team thinks about verification
by dcss_gardener on Tuesday November 17, 2026 @10:05AM
(#50532354)
It's really interesting to me to contrast the code and publications of deepseek and anthropic over the last year. Anthropic's stuff has some ideas that can be extended or applied but everything they do assumes basically unlimited frontier model inference and so is nearly unusable to me in the form they publish it.
Where deepseek prices inference the way systems programmers price latency. As a resource to be spent carefully towards specific goals, with systems in place to ensure you're within budget and getting the value you expect from it.
And then this also has second order effects on how each company treats human attention. Having fable xhigh do 3-4 passes over a change can legitimately reduce the need for human attention on it in a way that deepseek flash just can't really. So then they price operator attention in a similar way. With ant nearly considering human intervention a failure mode, while ds again prices it as a budget to be designed within.
And neither one is "right" speaking generally. I have my aesthetic preferences and my own constraints that make one a more appealing approach personally. But both are building towards the goals they have with consideration of the resources at their disposal, it's good engineering either way.
Re:How the Claude Code team thinks about verification
by radlad on Tuesday November 17, 2026 @10:52AM
(#50533129)
Yes. Anecdotally, I saved a ton of tokens (and bigger: time!) switching from having Claude drive Android emulator VMs (even with detailed skills and helpers) to using Roboelectric tests.
Re:How the Claude Code team thinks about verification
by bismanawaz on Tuesday November 17, 2026 @12:01PM
(#50534447)
Full app testing by agents would mean that each agent needs its own running app and browser use. So machines would cost too, not just the tokens in my opinion.
Re:How the Claude Code team thinks about verification
by oceanplexian on Tuesday November 17, 2026 @12:08PM
(#50534565)
> This article feels like the result of having unlimited tokens at your disposal.
Gong to be honest where do you think this is going?
Do you think tokens are going to magically become more expensive or models are going to become less efficient?
Of course you should be tokenmaxxing, you can already tokenmax with a model that competes with the SOTA 1 year ago on a moderately powerful laptop you can walk out of an Apple store with.
Re:How the Claude Code team thinks about verification
by throw10920 on Tuesday November 17, 2026 @12:21PM
(#50534804)
To be a bit more grounded and expand on your point:
I, personally, have already experienced situations where I was employing cost-optimization optimization strategies for models 8 months ago, that are borderline-useless now because the Pareto frontier has advanced both more than I thought and also in different ways than I thought (cheap models got smarter, better at following instructions, and more clever about tool calls).
That is, the strategies decreased in effectiveness disproportionately to how much the model improved.
It's worth considering that strategies that seem insanely expensive now may be economically viable in a year or two. Or possibly significantly sooner than that.
It's also possible that, like how jet engines get more efficient as they get larger (leading to the 737 MAX incidents, which seems spiritually significant) corporations with budgets orders of magnitude greater than ours benefit from very different strategies than we do.
Re:How the Claude Code team thinks about verification
by ar_lan on Tuesday November 17, 2026 @12:30PM
(#50534975)
This seems like a very strange comment to come from someone I am assuming is a SWE. When have we, ever, been happy with non-deterministic, objectively more expensive, solutions?
You're assuming a lot of costs that should just be considered ok in your stance:
* Why pay anything multiple times when you can replicate deterministically for free?
* Why require always-online models when you can run computation locally?
* Inversely, why use power-hungry/PU-hungry models locally when you can have a deterministic software that can be optimized to not* be memory/PU/power hungry to do validation?
I think tokenmaxxing can be a great start* to get things going but when it comes to productizing I want determinism and cheap optimizations, and using frontier models to build these cheap optimizations is what we've been doing at our company over this past year to great success.
Re:How the Claude Code team thinks about verification
by giancarlostoro on Tuesday November 17, 2026 @03:41PM
(#50537884)
Yeah, people don't realize what terrifies me most about stuff like Mythos, the employees can throw "unlimited" compute at it, to the point where it could go off and do something off-rails that they cannot react quickly enough to prevent. At what point is any of this okay? Honestly, I hope I'm wrong and it never happens.
Re:How the Claude Code team thinks about verification
by thefunnyman on Tuesday November 17, 2026 @08:44PM
(#50540834)
Yeah I work at a place that has high token budget and I see this nonsense all the time. Even putting cost aside it's crazy to me to introduce nondeterministic checks into your process. Surely we've all dealt with the pain of flaky tests before.
AI is a huge force multiplier and makes it almost trivial to create things like lint rules you'd otherwise probably not bother to set up. Use it for that instead of trying to get it to be a shitty linter itself.
Re:How the Claude Code team thinks about verification
by pixelready on Tuesday November 17, 2026 @09:35PM
(#50541145)
Not only is their token budget effectively unlimited, but they have a strong incentive to both discover more uses for their models when dogfooding and to encourage readers to consume more tokens. If all you’ve got is a hammer…
Re:How the Claude Code team thinks about verification
by FartyMcFarter on Tuesday November 17, 2026 @06:49AM
(#50530192)
They poached a lot of people from companies like Google that have solid engineering practices. I imagine it's a culture issue if anything, not a knowledge/background issue.
Re:How the Claude Code team thinks about verification
by thatsabadlook on Tuesday November 17, 2026 @07:24AM
(#50530464)
Googles engineers are good in those types of environments. Most environments aren't Google like at all. Ive seen so many small and medium companies hire googlers and burn money because of it. I actually watched a start up die because of a few key middle management/staff engineer hires from Google. They just could not operate for anyone other then themselves and were oblivious to the scale they worked on.
Re:How the Claude Code team thinks about verification
by koolba on Tuesday November 17, 2026 @06:53AM
(#50530221)
> Nobody can read every change, so we use failures as one way to decide what needs a person.
Whether a human can or should review every diff is a conscious project level decision. I have yet to see a project where it is not feasible. The trick is to break things into small enough pieces that you can review them. Just like one would do if you were doing such changes manually. Nobody in their right mind performs non-mechanical changes of 100s of files.
Re:How the Claude Code team thinks about verification
by fhd2 on Tuesday November 17, 2026 @07:07AM
(#50530330)
The "right mind" of companies is a bit different from that of individuals. Generally they care about three things: Revenue, costs and risks.
To companies, some risks are worth taking, if it can increase revenue or lower costs. Many companies happily do not-quite-legal things, taking into account the potential of being caught and the size of the fine.
In the case of Anthropic, from that perspective, it almost seems like a no-brainer that they would want to push an "our AI can do it all, no humans needed" narrative in everything they do. They need hype like we need oxygen.
Re:How the Claude Code team thinks about verification
by thefunnyman on Tuesday November 17, 2026 @08:52PM
(#50540895)
> Nobody can read every change, so we use failures as one way to decide what needs a person.
This keeps coming up at work and I feel like everyone's forgotten that code is cheap to produce and expensive to maintain. If you have devs assisted by agents vomiting out giant unreviewable PRs isn't that a sign that something's deeply wrong? It's always been the case that you can hack together something that appears to work but has deeper issues upon inspection yet there seems to be a massive push to just crank out the first version of whatever passes the bare minimum validation.
Re:How the Claude Code team thinks about verification
by Narann on Tuesday November 17, 2026 @06:53AM
(#50530226)
The more I read articles like this, the more I realize Claude is good for process prototyping, but you need to "bake" the process with flat code/script/organization to decrease costs.
The future of IA would actually be to large scale prototype easily, even more like Python was a prototype language.
Re:How the Claude Code team thinks about verification
by FartyMcFarter on Tuesday November 17, 2026 @06:56AM
(#50530251)
> Flaky tests get the same treatment. Claude can't skip them, add sleeps or retry until they pass. If it can't find the root cause, it deletes the test and writes down why.
If the tests are hermetic, you should always be able to find the root cause (by bisecting to determine when the test became flaky).
If the tests are not hermetic, it should still be possible to identify the dependency that is making them flaky, and use that as a justification if it still makes sense to delete the test.
Re:How the Claude Code team thinks about verification
by octoberfranklin on Tuesday November 17, 2026 @07:39PM
(#50540338)
Nothing is ever perfectly hermetic.
Unless your code is 100% single-threaded you'll always have variable timing between threads (influenced by things like caches, etc).
Tests need to be not just hermetic but also deterministic. The problem is that determinism doesn't have a glib "just wrap it in a sandbox bro" way to achieve it.
Re:How the Claude Code team thinks about verification
by FartyMcFarter on Wednesday November 18, 2026 @09:03AM
(#50546461)
> Nothing is ever perfectly hermetic.
Strictly speaking that's true, since there's hardware involved.
But if you introduce reasonable assumptions such as having non-faulty hardware on the vast majority of test runs, most unit tests are in fact perfectly hermetic and deterministic for all practical purposes.
Unit tests are usually single-threaded too, which cuts off a huge source of nondeterminism. Integration tests can quickly get more messy though.
Re:How the Claude Code team thinks about verification
by hliyan on Tuesday November 17, 2026 @07:05AM
(#50530314)
In my opinion, the output of a stochastic process can only be truly verified through a deterministic process. Tests already do this for functional accuracy. Linters can do this for line-level code, with code treated as text. But there is currently no such method of deterministic verification for the structure of the entire program, or more importantly the viability of the architecture when you throw in connectivity and IOPS.
Perhaps we need a level above linters: a tool that constructs the abstract syntax tree and verifies its various branches against deterministic rules.
Re:How the Claude Code team thinks about verification
by chamomeal on Tuesday November 17, 2026 @09:10AM
(#50531610)
> a tool that constructs the abstract syntax tree and verifies its various branches against deterministic rules
Isn’t that basically a linter?? If you go too far outside a regular ol’ linter (even a real nice type aware one like typescript-eslint) you hit a ceiling where some intelligent being needs to interpret things.
Though I will say I have seen interesting projects in clojure world like rewrite-clj, which makes it really easy to do structural analysis and modification of clojure programs. I’ve definitely thought there could be more room for stuff like that in the age of LLMs. I’ve also made more use out of ts-morph than ever before.
Also in my box of purely delusional ideas, I’ve thought about how cool it would be to encode LLM-derived semantics at the AST level, which then could be run through a regular deterministic linter. Like a switch case or pattern match could have “this is the happy path” or “this is an edge case” encoded into each branch. Then you could do stuff like collapse all non-happy-path lines, or highlight all edge case handling.
Re:How the Claude Code team thinks about verification
by Dashadower on Tuesday November 17, 2026 @12:59PM
(#50535442)
Its called formal verification. If you do try a refinement proof against a specification the proof architecture is just following the implementation's code and performing case anslysis for each branch possible.
The overall process that I am using does use a lot more time and AI spend than just asking AI to do something. I still do the latter for simple or noncritical changes. I had to switch to using DeepSeek as my workhorse to keep down my spend.
Re:How the Claude Code team thinks about verification
by fg137 on Tuesday November 17, 2026 @08:02AM
(#50530800)
I have stopped reading these articles/watching such videos because I already know what they talk about and how we are never going to adopt it. At my $dayjob, we don't use # of PR to evaluate someone's performance, we actually care about quality a lot, and we don't have >15k open issues in our project despite aggressively closing them (e.g. auto close issue after inactivity without confirming if the issue still exists) [0]. Well, it is also true we don't have unlimited tokens to waste.
Re:How the Claude Code team thinks about verification
by saagarjha on Tuesday November 17, 2026 @08:19AM
(#50530969)
This is just a depressing read, and I’m not even sure why the team decided to post it because it just looks horrible. Like, you’re telling me that the team’s approach to verification is that they post Claudeslop blog posts talking about the most unimaginative ways to write tests? That their idea of making things work is this?
> You can also close the verification loop by using the results to improve the agent’s performance on that task. For example, if an agent repeatedly fails a test that checks logs for user data, you can create a hook or a more explicit rule in your CLAUDE.md to avoid future issues. This (mostly) stops issues from happening again.
There are people out there who are doing all sorts of crazy neat things in the verification space using the tools you build and yet the people who could be helping seem to have no good ideas on how to help :/
Re:How the Claude Code team thinks about verification
by titzer on Tuesday November 17, 2026 @08:20AM
(#50530982)
I see a lot about adding skills to make Claude test things.
How about making Claude generate scripts that you can run?
Algorithms are better than agents. In fact, the whole point of AI is to reduce thinking to algorithmics. I didn't see a lot of evidence that the team understands this. If you need a magic box to test your other magic box, you have magic boxes all the way down.
Re:How the Claude Code team thinks about verification
by edg5000 on Tuesday November 17, 2026 @11:06PM
(#50541749)
LLMs will write a lot of tests by default, but anybody doing any kind of engineering knows that automated suites are just one layer of testing. There is always that manual layer, which LLMs can now help with. Even for hardware, they can control bench equipment and run through tests that can't be simulated (they are simulated, but you'd still want a final test on real hardware).
"skills" is just text files, you just have docs in your repo outlining the manual test procedures which you'd otherwise do yourself.
So I'd say you need both. Scripts plus manual testing, either done by you or the LLM.
Re:How the Claude Code team thinks about verification
by _nalply on Tuesday November 17, 2026 @08:47AM
(#50531319)
Verification? That's not so simple. Essentially humans must do it, or do it with an agent on a tight leash.
When working with an agent, I encountered two failure modes.
1. Automated testing. Once the agent proposed a complicated oracle. Because I didn't understand it, I thought, okay, whatever. But the oracle was wrong.
2. Reporting. The agent lied while writing the report. Once the agent said it's doing "exec app args", but it was "exec /bin/sh -c helper-script app args". That's just not the same.
Generally it amounts to "Quis custodiet ipsos custodes?" - this means verification is difficult, because you need to verify the verification. Even if you use a second agent to verify the verification, you don't know whether the second agent is doing what you mean.
Do what I mean is slippery with agents.
Currently I solve this problem by careful and strict with rules and asking many questions and if I don't understand something, I assume that the agent might have done something different than I expect.
Re:How the Claude Code team thinks about verification
by asmvolatile on Tuesday November 17, 2026 @10:30AM
(#50532733)
The Claude code team doesn’t think about jack sh*t. Interacting with Anthropic teams makes it clear that nobody at that company has any idea how their software works.
Re:How the Claude Code team thinks about verification
by ausbah on Tuesday November 17, 2026 @12:28PM
(#50534944)
i mean yeah, they’re very public that no one writes much less reads any code anymore. everything they do is via thousands of agents and burning quadrillions of tokens. their blogs and other public facing comms are full of “best practices” reiterating the above and how much more code their pushing everyday
most of that is prob for internal tooling and model related work, but seeing their public facing software makes me think that all this work hasn’t really translated to all that much more than 3 years ago
Re:How the Claude Code team thinks about verification
by redanddead on Tuesday November 17, 2026 @05:00PM
(#50538837)
Half of what they do is figure out how to break their own products lmao
the claude mac app keeps yoyoing every 6 months from cowork to claude code its the dumbest AI mac app i've ever used. I was helping my girlfriend set it up for work.
They added these idiotic restrictions like limiting computer use in cowork/normal_claude and asking you to pay to use it, I simply added a computer use mcp. Retarded.
Re:How the Claude Code team thinks about verification
by sashank_1509 on Tuesday November 17, 2026 @10:42AM
(#50532914)
Take 10-20 skilled programmers, the kind that made web browsers, game engines etc. Give them a year. They will end up with a superior working version of Claude Code without thousands of bugs and issues, that keep getting fixed and unfixed and trillions of tokens spent by AI.
They won’t even do all this verification nonsense and their end product will be of superior quality.
Re:How the Claude Code team thinks about verification
by tshaddox on Tuesday November 17, 2026 @10:49AM
(#50533072)
Is Claude Code considered to be a reliable piece of software these days?
I stopped using as my daily driver around 9 months ago, and at that time it felt very heavy/janky and seemed to have very regular bugs and breaking UI changes.
Re:How the Claude Code team thinks about verification
by tshaddox on Tuesday November 17, 2026 @08:36PM
(#50540767)
When I stopped using Claude Code exclusively, I experimented with pretty much everything that I heard about. Opencode, pi and omp, crush, Cursor CLI, Codex, probably several I’m forgetting. I always come back to OpenCode and end up using it the most.
More recently for non-work stuff I’ve been enjoying T3 Code, which abstracts over the individual agent CLIs and has a solid iOS app.
Re:How the Claude Code team thinks about verification
by kevincox on Tuesday November 17, 2026 @10:58AM
(#50533220)
My experience of Claude Code is a buggy mess. The only thing that seems to reliably work is the main chat (when they model APIs are online, but that is a separate topic) all of the other features such as file browser, diff viewer, model switcher, and whatever else I didn't think of seem to be working only on occasion.
I definitely wouldn't take their word on verification as a particularly good source.
Re:How the Claude Code team thinks about verification
by jrflo on Tuesday November 17, 2026 @11:38AM
(#50533995)
The desktop app is actually great now, I moved to it from the CLI recently. It has a live view of the context window and what is filling it, integrated terminal, diff, markdown viewer, file tree, browser, etc. If you're embracing agentic development, it's a great environment. The only bug I've encountered is sometimes it lags for ~1s when switching to a long conversation.
Re:How the Claude Code team thinks about verification
by ellyagg on Tuesday November 17, 2026 @12:18PM
(#50534756)
Reliable enough. Take care not to conflate reliability with overall value. Reliability is one metric, and easy to misread, like all individual metrics. Claude Code is very powerful and valuable overall.
Re:How the Claude Code team thinks about verification
by couchdb_ouchdb on Tuesday November 17, 2026 @12:33PM
(#50535013)
I still prefer the terminal. I used the UI the other day, and it was like really really verbose compared to the terminal filling up the screen with JIRAs and all sorts of things.
Re:How the Claude Code team thinks about verification
by anshumankmr on Tuesday November 17, 2026 @12:10PM
(#50534595)
Honestly, the amount of bullshit unit tests Claude keeps pushing nowadays. I took a few hours the past few weeks, some with AI, some by myself to hand remove some really dumb fucking test cases. God knows how long the CI must be running when these guys push their code.
Re:How the Claude Code team thinks about verification
by AlienRobot on Tuesday November 17, 2026 @01:09PM
(#50535623)
I wonder what was it even trained on?
Who ever had the time and will to write unit tests that mock every single function/class being called, checks the strings of thrown errors, etc., for every single function, for several tests per function?
The instant you change the implementation the tests break because it's testing how many times each method has been called of an object that was passed as argument.
That's just an insane way of writing tests. But for AI that's normal for some reason.
It feels like it's making unit tests the "textbook" way that in practice you wouldn't do because in practice half the things it's testing are inconsequential.
Re:How the Claude Code team thinks about verification
by bmitc on Tuesday November 17, 2026 @07:36PM
(#50540322)
Claude Code will often just test that data types, like structs or dataclasses can just have their fields read and written. It's infuriating. I don't let Claude write tests anymore without strict supervision.
Re:How the Claude Code team thinks about verification
by holoduke on Tuesday November 17, 2026 @12:31PM
(#50534986)
What we need are ultra fast testing suites. Specially e2e tests are notoriously slow. Runtime is big and its often difficult to isolate just a few tests. I want 1000 e2e tests in a few seconds instead of 10 minutes. It's also a mayor blocker for an agent that cannot work fast because of the constant waiting. At the same time you need the test feedback loop in order for the LLM to do proper shotgun programming.
Re:How the Claude Code team thinks about verification
by VadimPR on Tuesday November 17, 2026 @02:07PM
(#50536550)
I'm surprised by the negative feedback here. I've shipped quality software thanks to Claude, with a lot of features, bugfixes and performance improvements at velocity that is incomparable to doing it without an AI that has the stakeholders VERY happy.
The QA workflow I worked out is pretty much mirrored in this article, so speaking from the real world, this is the advice to follow. I'm on an Ultra plan for this work.
Re:How the Claude Code team thinks about verification
by jwpapi on Tuesday November 17, 2026 @03:19PM
(#50537592)
My main issue right now is that more verification would be more test runs, but running evals can get expensive easily. The only part I can’t really verify is UX (partly I can) and agent workflows itself.
The rest of stuff I barely care anymore. To me my current setup it just works.
Re:How the Claude Code team thinks about verification
by preordained on Tuesday November 17, 2026 @03:42PM
(#50537897)
If this didn't say Claude, even potentially it could be largely rewritten not to deal with LLMs...under those circumstances this would be seen as a pretty decent article about testing quality.
The projection of some hurt feelings (or actively hostile actors?) in these comments is kind of insane.
Re:How the Claude Code team thinks about verification
by Aperocky on Tuesday November 17, 2026 @04:28PM
(#50538447)
Reading this makes me better able to understand how claude code was an absolute memory hog on the computer.
For all of it's checks, there are not one mention about simplicity, conciseness or architectural coherence. And when you don't have your AI agents care about those things when you run them by the hundreds. You'll get spaghetti that works on a surface level, for now.
Re:How the Claude Code team thinks about verification
by causal on Tuesday November 17, 2026 @04:46PM
(#50538670)
So spin up more agents to write tests and review code? Come on.
The problem is rarely whether or not the code works: it's whether the agent really understood the problem and built the right thing. It can write perfectly good verification for the wrong problem.
For any non-trivial app it can take a deep domain expert days to determine whether the agent really understood the problem. I don't see that addressed at all.
Re:How the Claude Code team thinks about verification
by jeffybefffy519 on Tuesday November 17, 2026 @05:08PM
(#50538938)
I feel the ironic thing about this end-to-end development style with no human in the loop method that a lot of shops are going for is that it really places a lot more emphasis on manual user acceptance testing.
Otherwise the type of engagement loop that you get from just clauding every feature end to end pushes you to drop important quality steps because the agent is covering it but often it just isn't and you get slop with regressions everywhere.
TLDR we think of the frontier as "the best"... but really its the most unstable in this new world - Claude Code, Codex and their supporting models absolutely show this time and time again.
Re:How the Claude Code team thinks about verification
by brunooliv on Tuesday November 17, 2026 @06:58PM
(#50540014)
It’s definitely a very convenient way of phrasing that “an agent needs a way to verify it’s own work” and then it’s just defaulting to ridiculous markdown file conventions that are clearly made up (REVIEW.md? Seriously? Is that in the system prompt now? Does the code review skill burn a few extra thousand tokens hallucinating one?) and are both extremely fallible and expensive when much better alternatives existed.
If anything everything written in this post is the opposite of verification as it wraps deterministic tools around a totally random LLM layer. What’s the industry gone to?
Re:How the Claude Code team thinks about verification
by edg5000 on Tuesday November 17, 2026 @11:10PM
(#50541771)
It's clear that they throw a lot at the wall, but only 1 out of 10 features are really needed. Plan Mode for example is just silly. It just adds friction. Of course you plan, but the harness doesn't need a specific function for that. Of course you have review specs, but that doesn't need a standard.
9 out of 10 of these Anthropic features won't be copied by competitors and will eventually be dropped in future releases, if they stay around long enough. Remember those "Tip of the day" dialogs? Silly and gone.
Their actual LLM is still a beast, despite the rediculous things in the system prompt. Actually stripping the claude code system prompt might make the model perform better. Not sure if they still tolerate that, maybe they block that server-side now.
I am more interested in the remaining 10%.
“Also Claude, but we're fixing up the attribution to be super confident before claiming it”
https://x.com/bcherny/status/2111207278653399460?a=CQxYPj8Va...
Im already spending my $2k usd per month in tokens with just one or two agents. How do they expect me to do that?
We are an enterprise and thus have to use API tokens, not subscriptions
I have worked in companies that spend millions a month on some random infra. I'm obviously not going to complain that its not sustainable?!
If they spend tokens like Anthropic suggests, they won't be profitable for long.
If you want the fraudulent route:
Spin up a sub company, invent some paper employees, spend $2k on "Teams" seats ( ~40 seats? ) for them.
In my experience, each seat can run up 5-10 parallel agents within the monthly usage.
That's 200-400 agents for similar money. It requires some fraud or willingness to hire cheap labour to fill the seats.
On a more serious note, I genuinely feel bad for any company too large to take advantage of the teams plan, I've never seen a more generous subscription offering in my life when compared to API pricing and it still shocks me that enterprises are forced onto API pricing, but I guess it's heavily subsidising SMEs and therefore possibly helping the economy by funding innovation through more nimble businesses.
If you want Shakespeare, you need to trend closer and closer towards infinite agents, duh.
Did you forget that they literally publish benchmarks that show how much of the task can be done independently?
Starting with infra evidence and soon expanding to application verification loops. This is a very exciting place to build and we're seeing great results once the verification loop is closed for the teams
this u?
"The resource at “https://claude.dev/_next/static/chunks/241gnzjfqbp9f.css” preloaded with link preload was not used within a few seconds. Make sure all attributes of the preload tag are set correctly."
There’s a long section in the middle where the article describes per-change verification via skill files and “ask Claude to use the chrome-for-Claude tool” when checking if changes have introduced regressions.
But there’s already tools which do this without spending tokens tokens. Playwright will achieve equal quality but deterministic results across the same verification attempts. It is write-once-use-often, so has a marginal cost near zero.
I think future AI cost will be as cheap as doing an API call in classic sense.
In the future when AI is able to do everything, there will not be one single super-AI that does it all, but a whole ecosystem of different AI. This is the same as how in the biological world, while there are various degrees of generalists, there's no single cell, plant, or animal which does absolutely everything. If our body needs more blood, it makes more blood cells, it doesn't repurpose intestine or skin for the task.
Similar argument in economics, comparative advantage: an AI which can do everything is best used for the things which only it and no other AI can do, and leaving to other (smaller, cheaper) AI relatively simple tasks like face recognition.
Even where AI can do this, that doesn't mean the best solution is AI rather than a traditional API.
https://en.wikipedia.org/wiki/Bitter_lesson
First, last I checked AlphaZero is a much better chess player than any LLM, and by a large margin.
Second, this isn't even about finding the best model for small tasks, it's about best use of finite resources, compute or energy, which means that it still holds even if a future AI (call it Alice) did everything better than AlphaZero does chess, it would still make sense for Alice to offload chess to a merely-good model like AlphaZero so that Alice could spend more time doing the things that nothing else besides Alice could do:
https://en.wikipedia.org/wiki/Comparative_advantage
Small, specialized models can still benefit from the Bitter Lesson in that they can take advantage of ever-cheaper computation. They don’t contradict the lesson.
So no way it will be as cheap. That is just wishfull thinking.
On some level you are right, LLMs and the agents built with the various models and architectures will get cheaper. But there are some significant drawbacks by introducing this risk, regardless of cost. The future will need the capacity to understand this too.
This is the process of increasing efficiency today, automate manual processes to reduce human effort. Why wouldn’t models get better at this as well?
Not that it is an explicit, conscious direction, but it very much is one.
You’re not wrong, in that free flowing tokens are enough to convince such a group it’s simply good advice.
The more you use product X the more product X you need to buy. Nice to be selling product X.
I'm not working for a manager, so I don't need to use it at all.
It can meaningfully “type” faster than you.
Take the handicap if you really wish to, I suppose.
I still use LLMs (cautiously) for searching and troubleshooting API things.
I’m banking on that being an evolving, but necessary skillset. For a while at least. And I’m still gaining experience in the process.
Plenty of exposure, plenty of prior experience, it’s a weird mix of management-ish context handling and sme mentorship while also providing the research benefits you mentioned.
Same here. But as opposed to the "use AI" vs. "don't use AI" crowds, I'd prefer a more pragmatic approach:
Do whatever possible to maximize the quality of the software. And with quality I mean correctness, reliability.
AI is just another tool there. Where it helps produce more correct, reliable code, consider using it. Where it doesn't, don't.
Sadly it seems many companies have a different definition of "better" (featuritis, shortest time to market, developer productivity, etc).
A key pillar was introducing things like static analysis to provide cheap deterministic checks as the first goal and only then moving to other forms of verification.
Sure you can have linting, compilation, unit tests, e2e tests. Do you really think Anthropic doesn't do this?
In particular constructing ( and often deconstructing on change ) differential tests is very expensive relative to cheaper things to set up.
I felt this article's argument was poorly presented because of this seeming contradiction. The video I linked was much clearer about what works and what doesn't when scaling to inhuman levels of PR merges.
I also know of companies that run very expensive test suites using spot instances in AWS that cost around $100 each time. No one complains about it.
I think all the posts complaining about expenses forget to compare the orders of magnitudes.
As soon as Oracle Tests fail, the LLM has three options:
1. Rewrite the test - This undoes the benefit of the test in the first place 2. Implement workarounds and quirks to match the oracle, often at the expense of recreating buggy behaviour 3. Implement both corrected behaviour and quirks-mode behaviour to meet the oracle, raise further jobs to formalise the switch over and change of behaviour.
( And this "change of behaviour" might be a few pixels out of place! Or it could be a much more significant change. When trying to meet a goal, LLMs will often treat these with the same zeal. )
If you're working interactively, you can catch it and steer it to what's best for the feature. If you're fully automating development, then you're at the mercy of whatever the model does at that moment.
Oracle-based testing has its place, but I don't think it's good for large scale automated testing.
Also, you're talking about $500/release 2-3 times a week. That's fine, that's less than the cost of an employee.
Anthropic and others are talking about how to scale to merge 2000-3000 PRs per week.
At that scale, that cost per merge is $6M/month, surely a cost you'd want to minimise if you can cut them by half with better type-checking, static analysis, and other cheap deterministic tests, even if that's still a small cost of the whole project.
If nothing else, it's pure waste. Our planet is (mostly metaphorically) burning, and I believe that more efficient IT processes and development practices would go a long way to helping mitigate some of that.
https://kenrahn.com/JFK/Critical_thinking/What_do_many_piece...
If one’s original argument doesn’t have enough weight, they just add other weak ones to justify it.
Anyway, responding to the strongest version of it: the economics of any new system always looks bad before it becomes popular. I heard people say the same thing about AI driven coding and now there’s absolutely no going back. If you were a software engineer living in 1990s and you heard someone say that Uber, a ride sharing company, spends 100s of millions per year on just microservices and db infra, you’d be skeptical again.
Usually the costs justify themselves because of the productivity it brings. People spend 10x on cloud for this reason.
If you think I’m wrong, let’s make a bet of sorts. I’m highly confident that AI driven testing will be another normal and extensively used tool just like how CI/CD as a concept is today
Most jobs I worked in wouldn't bother discussing 6k of expenses. The meetings and decisions making involved probably cost more
That seems like something to pay attention to, even if you prefer to treat it as a cautionary tale with your popcorn at the ready for when you expect it all comes crashing down.
It doesn't take 5 minutes to prompt claude, "Please pick up Ticket #1234567".
With modern models, you rarely need much more than that. Perhaps checking in an hour later with a quick prompt, "Progress report?" if it's still working on it.
Claude having written the ticket in the first place, defined acceptance criteria. Perhaps a human has skimmed it for a go/no-go decision, perhaps not.
The point is that at that scale you can't humanly read the code, so you need to define verification strategies that work for the machine.
Precisely so that you're not just running on pure vibes.
Why is that scale needed in the first place?
With my skeptic hat on, it'll be interesting to see what happens when you look at their internal speed, in PRs/month vs the outward velocity in features and improvements actually shipped.
With my booster hat, it's something to learn from so I can better scale from juggling hand-reviewed claude sessions to a future it might be demanded I join at some point.
We're in a weird space in development, where the initial mechanical work that used to take a week now takes an hour at most, but what remains is a lot of manual verification and developer-testing.
If we can learn from what these mega-swarms are doing to reduce the need for manual developer testing ( because at that speed it's clear there must be close to none ), then that is a useful lesson that might reduce our work per task too and let us do more.
And yes, some of that may be stuff we've known for years, such as functional CI/CD, better linting, better automated e2e testing with playwright, etc.
But actually convincing management to invest in that kind of thing isn't always easy, especially if it risks instability of a process that from their perspective has been working fine for decades, even if that "process" is someone logging onto a server to manually fire off a script that was last touched 15 years ago.
So having talks such as this to point to is useful. You can show management an expert on a stage, and in some companies an expert on a stage can be weirdly convincing to management in a way that their own employees sometimes are not.
I don't think the low speed of development was an actual bottleneck. Coding in the real world is actually a mix of mechanical typing and information gathering. The typing part is not that relevant as it's actually not much time to type in 500 lines of code (if you touch type) assuming that's the median size of PR. It is the gathering of information that is IMO crucial for the health of the software.
It's like robots in a chemical lab. You can probably build some mechanical apparatus that can mix whatever components you want. But you still need a scientist to design experiments and to check the result. And while you can help speed up some of the process, what's valuable is the knowledge that the scientist extract, not the amount of experiments being done.
This scale of work is only possible because there's no information gathering going on. Instead it's just busy work with wishes and prayers hoping to get a good result.
I can't see how I would be able to keep customers happy without actually reading the code and understand what it is doing.
It sounds like you’re doing things basically the way they’ve always been done.
What sounds like that exactly? I don't see anything in the GP about what and how they're "doing things" one way or the other.
They're making an argument about someone's statements. Engage with the argument or don't, but don't make it weird like this.
This is so far from engaging with how Lauren actually works that’s it’s barely worth even responding to. They’re describing a standard software engineer workflow as it existed decades ago. Lauren is playing on a totally different level, more successfully I suspect (though who knows because while she’s public, this critic is anonymous).
Basically management I guess
Now, are those PRs doing useless work? Are they introducing unnecessary risks or trade offs? Are they building towards something impossible to understand or that will fall apart under its own sloppy weight?
Those seem more like the right questions to me, in the long run, and I suspect that’s what you really have in mind. Because if the answers are no, then why wouldn’t you be doing as many PRs as possible?
So to me "not micromanaging PRs" doesn't make any sense at all. Bragging about how many PRs you can get merged is like bragging about how many WPM you can get when prompting an AI instead of manually typing.
That comes with trade-offs, but some may be willing to accept those. It's ultimately the decision of the developers and enterprise how they want to build their product, can't really say anything else about this
All productivity metrics have flaws (usually deep ones). Doesn’t mean they’re not instructional though.
My PRs get assessed for if a human is needed. I could probably do that at the branch level, but PRs just has all the metadata and CI/CD tooling etc etc.
Depending on the domain, human review can be mandatory. (i.e. if you're a public company and the thing you're touching can materially affect the company finances)
This reminds me of when "The Joel Test" was first published and it was met by people unbelieving there was a world out there who weren't even using source control at the time, let alone a bug-tracker.
Love to find some high quality youtubes to listen too to reduce my screen time.
For me this article is more aimed at devs that are still stuck treating code generation and verification as two separate tasks. Giving agents the right tools (including playwright) to do both at once makes things more token efficient, not less.
Do you not see the propaganda through the pseudo helpfulness?
I find it amusing that people phrase bog-standard "what can our product do" documents as "propaganda", even when it's a dogfooding exercise.
Like, I guess? In the sense that all marketing can be seen as capitalist propaganda?
Scandalous.
The reason you should use Claude and not playwright always is because the the invariant you wanna capture is at the natural language level.
Like imagine I wanted to verify that Claude Code works and doesn't break. I can describe the invariant I want in natural language. Like: make sure that subagents don't block the original thread.
Doing this in playwright means that you have already bought into some previously existing architecture which means your tests are now correlated to your code. Having the invariants written in natural language allows you to decouple code vs the invariant.
I expect a lot of greybeard conservative programmers to keep opposing using AI here until eventually, not using AI will be seen as more risky.
That was kind of my point. Natural language almost always survives rearchitectures because your higher level interface to your end customer rarely has churn.
In a few years, agent driven testing will become normal and almost expected in some enterprise companies. Obviously we will still have people like you opposing it on grounds of “but just write tests in playwright”.
I also find the greybeard archetype fascinating. They’ve opposed almost everything new including Cloud, microservices, kubernetes. Now with AI. And here in this thread with AI driven tests. They seem to think programming should be done exactly how it was done when they were in the prime of their career. Anything before that is archaic and anything after is hype.
I’m saying that a default response to challenge AI testing by saying “just write normal tests” is the typical greybeard archetypical response. And they don’t really have a good track record considering they’d by default challenge cloud, kubernetes etc
If yes, that’s what I recommended.
I think tokens are near-free already, even for those of us outside Anthropic.
More importantly, your philosophy comes from a time where thinking had to be front-loaded: you try to predict everything that can go wrong, then design deterministic tests that can catch those things and a decision tree to figure out what to do in each case. This approach was the only viable way until now, but maybe that will change. Having the equivalent of a senior engineer thoroughly review each error with full context of what the application and feature is trying to achieve could perhaps uncover different insights, or cover edge cases more smoothly.
No real software engineer think that way, as things that can go wrong are near infinite. Instead you design the right thing and write tests that verify that it’s indeed the right thing and add some more that are the wrong thing, especially at the edge of definition. Then you rely on the dev to not make very obvious errors like coding something that is not in the requirements or is completely opposite.
In this age of AI, anything well understood and effective while being efficient is out. These tokens must be used, atmosphere must be heated.
It's November. Winter is coming.
Climate change and global warming is really a scam, they have made it snow with chemtrails in the 80s, it seems.
Though I miss snow and cold. They are not making seasons as they used to anymore. They shouldn't have privatized weather.
(This comment is written by a human, and contains an implicit /s)
right?
How long do you think that will last?
Am over 11 billion tokens this month on a subscription. In API usage its close to $5000 and there's still 13 days left in the month.
Eventually the music in musical chairs stops. Already many anecdotal reports of the always opaque limits being lowered across the board. On a gemini pro account for personal use I can use a weeks worth of Argon in about 20 minutes on agy.
Many benchmarks are already saturated today on many tedious tasks still being done by humans across the world. Frontier labs are introducing their own budget options prior to IPO, undercutting their revenue, because otherwise they risk losing seats.
Choosing what to do and how to do it is the actual frontier now and is still not apparently well supported in any model. Current so-called decision models are not decision models, they are available-fact-evaluating models.
Exactly my point, think the easy things won't even need 3rd party apis in the near future.
But frontier models only seem to need more and more VRAM to keep winning the race. Feel free to look up the cost of running Kimi K3 open weights today, probably the best estimate known publicly, it's hardly a casual affair.
Don't forget that it's a hypothesis, not a foregone conclusion, that this will stop. Part of the question is whether demand will continue to rise. If volume increases sufficiently, you can cut prices and improve profits at the same time, as long as you continue to make a profit at the margin.
Moore's Law kept Intel and competitors pushing R&D budget that pushed perf/watt year after year because the overall computing market just kept growing and growing. The growing demand soaked up everything Intel and co. pushed into the market; the R&D spend helped players retain market share in the face of competition. Why do you think AI is different? R&D into better, faster, cheaper chips brings down the cost of both training and inference over time, and cheaper token cost unlocks additional demand, which justifies more R&D budget, in a virtuous cycle. It continues until there's really no more demand to be unlocked, no more improvements can be made to the chips (i.e. you hit the laws of physics), or there's no more money to be invested (i.e. even capital markets have limits, with only so much cash in circulation, or macro interference like war).
Do we really have any reason to believe that demand will taper off?
> ● Agent "[REDACTED] audit" failed: Agent terminated early due to an API error: You've hit your 5-hour limit · resets 10:50pm (Europe/Berlin) (error type rate_limit, HTTP 429, request id [REDACTED], model sent to the API: claude-opus-5-5) > ● Agent "Verify [REDACTED] bug" failed: Agent terminated early due to an API error: You've hit your 5-hour limit · resets 10:50pm (Europe/Berlin) (error type rate_limit, HTTP 429, request id [REDACTED], model sent to the API: claude-opus-5-5) > ● Agent "Verify [REDACTED] bug" failed: Agent terminated early due to an API error: You've hit your 5-hour limit · resets 10:50pm (Europe/Berlin) (error type rate_limit, HTTP 429, request id [REDACTED], model sent to the API: claude-opus-5-5) > ● Agent "[REDACTED]" failed: Agent terminated early due to an API error: You've hit your 5-hour limit · resets 10:50pm (Europe/Berlin) (error type rate_limit, HTTP 429, request id [REDACTED], model sent to the API: claude-opus-5-5) > ● Agent "Verify [REDACTED] injection" failed: Agent terminated early due to an API error: You've hit your 5-hour limit · resets 10:50pm (Europe/Berlin) (error type rate_limit, HTTP 429, request id [REDACTED], model sent to the API: claude-opus-5-5) > ● Agent "Verify [REDACTED] regression" failed: Agent terminated early due to an API error: You've hit your 5-hour limit · resets 10:50pm (Europe/Berlin) (error type rate_limit, HTTP 429, request id [REDACTED], model sent to the API: claude-opus-5-5) > ● Agent "Verify [REDACTED] race" failed: Agent terminated early due to an API error: You've hit your 5-hour limit · resets 10:50pm (Europe/Berlin) (error type rate_limit, HTTP 429, request id [REDACTED], model sent to the API: claude-opus-5-5) > ● Agent "[REDACTED] root-cause" failed: Agent terminated early due to an API error: You've hit your 5-hour limit · resets 10:50pm (Europe/Berlin) (error type rate_limit, HTTP 429, request id [REDACTED], model sent to the API: claude-opus-5-5) > ● Agent "[REDACTED]" failed: Agent terminated early due to an API error: You've hit your 5-hour limit · resets 10:50pm (Europe/Berlin) (error type rate_limit, HTTP 429, request id [REDACTED], model sent to the API: claude-opus-5-5)
If you're doing this, the agents should create deterministic tests from its process rather than Agent tests or just writes deterministic tests for start. This is a bit of an absurd approach.
The issue I find with these articles is that they are basically the same boat as "Facebook Engineering" articles were 20 years ago.
People will be "oh because Anthropic does it it must be the way", then a bunch of influencers will retweet it as "anthropic kills writing tests", which some CEO's and team leads will parrot and get their team to absorb these practices even tho they are not the best practices.
Some of these things are just reinventing the wheel, like this is a basic script called "run_simulator.sh" and a pre-commit test hook, not a skill and "commit gate" that needs to check session transcript:
>To codify how to start the simulator or run your app, create a skill called verify. Put it in a .claude/skills/ folder in your repo, as close to the relevant code as you can, and add as many as you need. Claude knows to use the verify skill before committing changes.
>We've also created a commit gate that can block a commit if the session did not contain /code-review or /verify. We've set it up with off, observe, and enforce modes.
Where deepseek prices inference the way systems programmers price latency. As a resource to be spent carefully towards specific goals, with systems in place to ensure you're within budget and getting the value you expect from it.
And then this also has second order effects on how each company treats human attention. Having fable xhigh do 3-4 passes over a change can legitimately reduce the need for human attention on it in a way that deepseek flash just can't really. So then they price operator attention in a similar way. With ant nearly considering human intervention a failure mode, while ds again prices it as a budget to be designed within.
And neither one is "right" speaking generally. I have my aesthetic preferences and my own constraints that make one a more appealing approach personally. But both are building towards the goals they have with consideration of the resources at their disposal, it's good engineering either way.
I think it's worth designing around that because tokens will only get cheaper over time.
Gong to be honest where do you think this is going?
Do you think tokens are going to magically become more expensive or models are going to become less efficient?
Of course you should be tokenmaxxing, you can already tokenmax with a model that competes with the SOTA 1 year ago on a moderately powerful laptop you can walk out of an Apple store with.
I, personally, have already experienced situations where I was employing cost-optimization optimization strategies for models 8 months ago, that are borderline-useless now because the Pareto frontier has advanced both more than I thought and also in different ways than I thought (cheap models got smarter, better at following instructions, and more clever about tool calls).
That is, the strategies decreased in effectiveness disproportionately to how much the model improved.
It's worth considering that strategies that seem insanely expensive now may be economically viable in a year or two. Or possibly significantly sooner than that.
It's also possible that, like how jet engines get more efficient as they get larger (leading to the 737 MAX incidents, which seems spiritually significant) corporations with budgets orders of magnitude greater than ours benefit from very different strategies than we do.
You're assuming a lot of costs that should just be considered ok in your stance:
* Why pay anything multiple times when you can replicate deterministically for free? * Why require always-online models when you can run computation locally? * Inversely, why use power-hungry/PU-hungry models locally when you can have a deterministic software that can be optimized to not* be memory/PU/power hungry to do validation?
I think tokenmaxxing can be a great start* to get things going but when it comes to productizing I want determinism and cheap optimizations, and using frontier models to build these cheap optimizations is what we've been doing at our company over this past year to great success.
AI is a huge force multiplier and makes it almost trivial to create things like lint rules you'd otherwise probably not bother to set up. Use it for that instead of trying to get it to be a shitty linter itself.
Whether a human can or should review every diff is a conscious project level decision. I have yet to see a project where it is not feasible. The trick is to break things into small enough pieces that you can review them. Just like one would do if you were doing such changes manually. Nobody in their right mind performs non-mechanical changes of 100s of files.
We're in a brave new world, many many big name companies are doing this.
Not 6 months ago, sure. Today? It's not totally crazy. In a year it will be routine.
To companies, some risks are worth taking, if it can increase revenue or lower costs. Many companies happily do not-quite-legal things, taking into account the potential of being caught and the size of the fine.
In the case of Anthropic, from that perspective, it almost seems like a no-brainer that they would want to push an "our AI can do it all, no humans needed" narrative in everything they do. They need hype like we need oxygen.
This keeps coming up at work and I feel like everyone's forgotten that code is cheap to produce and expensive to maintain. If you have devs assisted by agents vomiting out giant unreviewable PRs isn't that a sign that something's deeply wrong? It's always been the case that you can hack together something that appears to work but has deeper issues upon inspection yet there seems to be a massive push to just crank out the first version of whatever passes the bare minimum validation.
The future of IA would actually be to large scale prototype easily, even more like Python was a prototype language.
If the tests are hermetic, you should always be able to find the root cause (by bisecting to determine when the test became flaky).
If the tests are not hermetic, it should still be possible to identify the dependency that is making them flaky, and use that as a justification if it still makes sense to delete the test.
Unless your code is 100% single-threaded you'll always have variable timing between threads (influenced by things like caches, etc).
Tests need to be not just hermetic but also deterministic. The problem is that determinism doesn't have a glib "just wrap it in a sandbox bro" way to achieve it.
Strictly speaking that's true, since there's hardware involved.
But if you introduce reasonable assumptions such as having non-faulty hardware on the vast majority of test runs, most unit tests are in fact perfectly hermetic and deterministic for all practical purposes.
Unit tests are usually single-threaded too, which cuts off a huge source of nondeterminism. Integration tests can quickly get more messy though.
Perhaps we need a level above linters: a tool that constructs the abstract syntax tree and verifies its various branches against deterministic rules.
How to do this for architecture: unsure.
Isn’t that basically a linter?? If you go too far outside a regular ol’ linter (even a real nice type aware one like typescript-eslint) you hit a ceiling where some intelligent being needs to interpret things.
Though I will say I have seen interesting projects in clojure world like rewrite-clj, which makes it really easy to do structural analysis and modification of clojure programs. I’ve definitely thought there could be more room for stuff like that in the age of LLMs. I’ve also made more use out of ts-morph than ever before.
Also in my box of purely delusional ideas, I’ve thought about how cool it would be to encode LLM-derived semantics at the AST level, which then could be run through a regular deterministic linter. Like a switch case or pattern match could have “this is the happy path” or “this is an edge case” encoded into each branch. Then you could do stuff like collapse all non-happy-path lines, or highlight all edge case handling.
After I created this skill for verification, it completely changed my level of trust with AI coding.
https://github.com/gregwebs/skills-sdlc/blob/main/skills/ver...
The overall process that I am using does use a lot more time and AI spend than just asking AI to do something. I still do the latter for simple or noncritical changes. I had to switch to using DeepSeek as my workhorse to keep down my spend.
https://github.com/gregwebs/skills-sdlc/blob/main/skills/imp...
More snake oil and theatrical agent setups described in insufferable Claudish. I think the target group of Claude is people with an IQ below 100.
[0] https://github.com/anthropics/claude-code/issues
> You can also close the verification loop by using the results to improve the agent’s performance on that task. For example, if an agent repeatedly fails a test that checks logs for user data, you can create a hook or a more explicit rule in your CLAUDE.md to avoid future issues. This (mostly) stops issues from happening again.
There are people out there who are doing all sorts of crazy neat things in the verification space using the tools you build and yet the people who could be helping seem to have no good ideas on how to help :/
How about making Claude generate scripts that you can run?
Algorithms are better than agents. In fact, the whole point of AI is to reduce thinking to algorithmics. I didn't see a lot of evidence that the team understands this. If you need a magic box to test your other magic box, you have magic boxes all the way down.
"skills" is just text files, you just have docs in your repo outlining the manual test procedures which you'd otherwise do yourself. So I'd say you need both. Scripts plus manual testing, either done by you or the LLM.
When working with an agent, I encountered two failure modes.
1. Automated testing. Once the agent proposed a complicated oracle. Because I didn't understand it, I thought, okay, whatever. But the oracle was wrong.
2. Reporting. The agent lied while writing the report. Once the agent said it's doing "exec app args", but it was "exec /bin/sh -c helper-script app args". That's just not the same.
Generally it amounts to "Quis custodiet ipsos custodes?" - this means verification is difficult, because you need to verify the verification. Even if you use a second agent to verify the verification, you don't know whether the second agent is doing what you mean.
Do what I mean is slippery with agents.
Currently I solve this problem by careful and strict with rules and asking many questions and if I don't understand something, I assume that the agent might have done something different than I expect.
we don’t actually do any of the work but we think about it
most of that is prob for internal tooling and model related work, but seeing their public facing software makes me think that all this work hasn’t really translated to all that much more than 3 years ago
the claude mac app keeps yoyoing every 6 months from cowork to claude code its the dumbest AI mac app i've ever used. I was helping my girlfriend set it up for work.
They added these idiotic restrictions like limiting computer use in cowork/normal_claude and asking you to pay to use it, I simply added a computer use mcp. Retarded.
They won’t even do all this verification nonsense and their end product will be of superior quality.
I stopped using as my daily driver around 9 months ago, and at that time it felt very heavy/janky and seemed to have very regular bugs and breaking UI changes.
I’d be happy to hear that it has gotten better!
More recently for non-work stuff I’ve been enjoying T3 Code, which abstracts over the individual agent CLIs and has a solid iOS app.
I definitely wouldn't take their word on verification as a particularly good source.
Who ever had the time and will to write unit tests that mock every single function/class being called, checks the strings of thrown errors, etc., for every single function, for several tests per function?
The instant you change the implementation the tests break because it's testing how many times each method has been called of an object that was passed as argument.
That's just an insane way of writing tests. But for AI that's normal for some reason.
It feels like it's making unit tests the "textbook" way that in practice you wouldn't do because in practice half the things it's testing are inconsequential.
The QA workflow I worked out is pretty much mirrored in this article, so speaking from the real world, this is the advice to follow. I'm on an Ultra plan for this work.
The rest of stuff I barely care anymore. To me my current setup it just works.
The projection of some hurt feelings (or actively hostile actors?) in these comments is kind of insane.
For all of it's checks, there are not one mention about simplicity, conciseness or architectural coherence. And when you don't have your AI agents care about those things when you run them by the hundreds. You'll get spaghetti that works on a surface level, for now.
The problem is rarely whether or not the code works: it's whether the agent really understood the problem and built the right thing. It can write perfectly good verification for the wrong problem.
For any non-trivial app it can take a deep domain expert days to determine whether the agent really understood the problem. I don't see that addressed at all.
Otherwise the type of engagement loop that you get from just clauding every feature end to end pushes you to drop important quality steps because the agent is covering it but often it just isn't and you get slop with regressions everywhere.
TLDR we think of the frontier as "the best"... but really its the most unstable in this new world - Claude Code, Codex and their supporting models absolutely show this time and time again.
If anything everything written in this post is the opposite of verification as it wraps deterministic tools around a totally random LLM layer. What’s the industry gone to?
9 out of 10 of these Anthropic features won't be copied by competitors and will eventually be dropped in future releases, if they stay around long enough. Remember those "Tip of the day" dialogs? Silly and gone.
Their actual LLM is still a beast, despite the rediculous things in the system prompt. Actually stripping the claude code system prompt might make the model perform better. Not sure if they still tolerate that, maybe they block that server-side now.
I got excited to see where they are on going beyond testing, but guess that means they are in the crawl phase.