News for hackers. Stuff that matters.
Re:How the Claude Code team thinks about verification by xnorswap on Tuesday November 17, 2026 @06:46AM (#50530167)
I agree completely, there was a much better talk on this by Lauren Tan: https://www.youtube.com/watch?v=O_zcc6Xl12g

A key pillar was introducing things like static analysis to provide cheap deterministic checks as the first goal and only then moving to other forms of verification.

Re:How the Claude Code team thinks about verification by simianwords on Tuesday November 17, 2026 @06:53AM (#50530230)
You are not agreeing with the OP. You are just saying that there are many levels to testing and each give their own swiss cheese level of verification.

Sure you can have linting, compilation, unit tests, e2e tests. Do you really think Anthropic doesn't do this?

Re:How the Claude Code team thinks about verification by xnorswap on Tuesday November 17, 2026 @07:06AM (#50530318)
I was agreeing that this page as presented felt like it was from the POV of people will unlimited tokens to burn.

In particular constructing ( and often deconstructing on change ) differential tests is very expensive relative to cheaper things to set up.

I felt this article's argument was poorly presented because of this seeming contradiction. The video I linked was much clearer about what works and what doesn't when scaling to inhuman levels of PR merges.

Re:How the Claude Code team thinks about verification by simianwords on Tuesday November 17, 2026 @07:18AM (#50530416)
I don't agree with "very expensive" claim. My friend works in a f500. They do releases like 2-3 times a week. Assuming each release uses $500 of tokens (higher estimate) just for verification, that's like $6000 a month and that's just a small percent over the tokens used for coding. What's the big deal? Their team brings in way more money than that - its a drop in the ocean.

I also know of companies that run very expensive test suites using spot instances in AWS that cost around $100 each time. No one complains about it.

I think all the posts complaining about expenses forget to compare the orders of magnitudes.

Re:How the Claude Code team thinks about verification by xnorswap on Tuesday November 17, 2026 @07:49AM (#50530702)
It's not the per-run costs I worry about, it's the costs of churn.

As soon as Oracle Tests fail, the LLM has three options:

1. Rewrite the test - This undoes the benefit of the test in the first place 2. Implement workarounds and quirks to match the oracle, often at the expense of recreating buggy behaviour 3. Implement both corrected behaviour and quirks-mode behaviour to meet the oracle, raise further jobs to formalise the switch over and change of behaviour.

( And this "change of behaviour" might be a few pixels out of place! Or it could be a much more significant change. When trying to meet a goal, LLMs will often treat these with the same zeal. )

If you're working interactively, you can catch it and steer it to what's best for the feature. If you're fully automating development, then you're at the mercy of whatever the model does at that moment.

Oracle-based testing has its place, but I don't think it's good for large scale automated testing.

Also, you're talking about $500/release 2-3 times a week. That's fine, that's less than the cost of an employee.

Anthropic and others are talking about how to scale to merge 2000-3000 PRs per week.

At that scale, that cost per merge is $6M/month, surely a cost you'd want to minimise if you can cut them by half with better type-checking, static analysis, and other cheap deterministic tests, even if that's still a small cost of the whole project.

If nothing else, it's pure waste. Our planet is (mostly metaphorically) burning, and I believe that more efficient IT processes and development practices would go a long way to helping mitigate some of that.

Re:How the Claude Code team thinks about verification by simianwords on Tuesday November 17, 2026 @12:06PM (#50534515)
It’s not a good sign that you had to bring up the environment to justify this..
Re:How the Claude Code team thinks about verification by nozzlegear on Tuesday November 17, 2026 @01:33PM (#50536027)
[delayed]
Re:How the Claude Code team thinks about verification by Rapzid on Tuesday November 17, 2026 @04:10PM (#50538232)
That's a weak man fallacy. The strongest parts of the argument are costs and efficiency.
Re:How the Claude Code team thinks about verification by simianwords on Tuesday November 17, 2026 @06:17PM (#50539599)
But the author also did a fallacy

https://kenrahn.com/JFK/Critical_thinking/What_do_many_piece...

If one’s original argument doesn’t have enough weight, they just add other weak ones to justify it.

Anyway, responding to the strongest version of it: the economics of any new system always looks bad before it becomes popular. I heard people say the same thing about AI driven coding and now there’s absolutely no going back. If you were a software engineer living in 1990s and you heard someone say that Uber, a ride sharing company, spends 100s of millions per year on just microservices and db infra, you’d be skeptical again.

Usually the costs justify themselves because of the productivity it brings. People spend 10x on cloud for this reason.

If you think I’m wrong, let’s make a bet of sorts. I’m highly confident that AI driven testing will be another normal and extensively used tool just like how CI/CD as a concept is today

Re:How the Claude Code team thinks about verification by cicko on Tuesday November 17, 2026 @08:03AM (#50530811)
Come back to Earth, please. Not everyone wants to suddenly spend $6K more per month on deployments.
Re:How the Claude Code team thinks about verification by tayo42 on Tuesday November 17, 2026 @10:37AM (#50532834)
People really work in different worlds.

Most jobs I worked in wouldn't bother discussing 6k of expenses. The meetings and decisions making involved probably cost more

Re:How the Claude Code team thinks about verification by theshrike79 on Tuesday November 17, 2026 @04:13PM (#50538272)
[delayed]
Re:How the Claude Code team thinks about verification by sceptic123 on Tuesday November 17, 2026 @07:17AM (#50530409)
If you take the post at face value they are saying some of those tests aren't valuable
Re:How the Claude Code team thinks about verification by rolosa on Tuesday November 17, 2026 @07:44AM (#50530647)
That developer brags about slopping out 2000+ PRs a month, how is anything they have to say useful?
Re:How the Claude Code team thinks about verification by xnorswap on Tuesday November 17, 2026 @08:12AM (#50530901)
Slopping out 2k+ PRs/mo while also claiming to reduce defect rates.

That seems like something to pay attention to, even if you prefer to treat it as a cautionary tale with your popcorn at the ready for when you expect it all comes crashing down.

Re:How the Claude Code team thinks about verification by rolosa on Tuesday November 17, 2026 @08:49AM (#50531344)
At 2000 PR/mo, assuming a "standard" 8 hour per day, 5 days a week workweek with no time factored for lunch, meetings, breaks... literally all they do for 8 hours a day, 5 days a week, is submitting PRs. That works out to roughly 5 minutes to slop, review, and document a single PR, approx 92 PRs per work day. That tells me they aren't doing shit.
Re:How the Claude Code team thinks about verification by xnorswap on Tuesday November 17, 2026 @09:09AM (#50531596)
I suspect that an ever increasing number of tickets being raised are being raised by agents themselves, nitpicking and being unable to deal with the idea of implementations having trade-offs.

It doesn't take 5 minutes to prompt claude, "Please pick up Ticket #1234567".

With modern models, you rarely need much more than that. Perhaps checking in an hour later with a quick prompt, "Progress report?" if it's still working on it.

Claude having written the ticket in the first place, defined acceptance criteria. Perhaps a human has skimmed it for a go/no-go decision, perhaps not.

The point is that at that scale you can't humanly read the code, so you need to define verification strategies that work for the machine.

Precisely so that you're not just running on pure vibes.

Re:How the Claude Code team thinks about verification by skydhash on Tuesday November 17, 2026 @09:34AM (#50531934)
> The point is that at that scale you can't humanly read the code

Why is that scale needed in the first place?

Re:How the Claude Code team thinks about verification by xnorswap on Tuesday November 17, 2026 @09:59AM (#50532270)
I don't think it is, but I think it's something worth paying attention to.

With my skeptic hat on, it'll be interesting to see what happens when you look at their internal speed, in PRs/month vs the outward velocity in features and improvements actually shipped.

With my booster hat, it's something to learn from so I can better scale from juggling hand-reviewed claude sessions to a future it might be demanded I join at some point.

We're in a weird space in development, where the initial mechanical work that used to take a week now takes an hour at most, but what remains is a lot of manual verification and developer-testing.

If we can learn from what these mega-swarms are doing to reduce the need for manual developer testing ( because at that speed it's clear there must be close to none ), then that is a useful lesson that might reduce our work per task too and let us do more.

And yes, some of that may be stuff we've known for years, such as functional CI/CD, better linting, better automated e2e testing with playwright, etc.

But actually convincing management to invest in that kind of thing isn't always easy, especially if it risks instability of a process that from their perspective has been working fine for decades, even if that "process" is someone logging onto a server to manually fire off a script that was last touched 15 years ago.

So having talks such as this to point to is useful. You can show management an expert on a stage, and in some companies an expert on a stage can be weirdly convincing to management in a way that their own employees sometimes are not.

Re:How the Claude Code team thinks about verification by skydhash on Tuesday November 17, 2026 @11:01AM (#50533275)
> We're in a weird space in development, where the initial mechanical work that used to take a week now takes an hour at most, but what remains is a lot of manual verification and developer-testing.

I don't think the low speed of development was an actual bottleneck. Coding in the real world is actually a mix of mechanical typing and information gathering. The typing part is not that relevant as it's actually not much time to type in 500 lines of code (if you touch type) assuming that's the median size of PR. It is the gathering of information that is IMO crucial for the health of the software.

It's like robots in a chemical lab. You can probably build some mechanical apparatus that can mix whatever components you want. But you still need a scientist to design experiments and to check the result. And while you can help speed up some of the process, what's valuable is the knowledge that the scientist extract, not the amount of experiments being done.

This scale of work is only possible because there's no information gathering going on. Instead it's just busy work with wishes and prayers hoping to get a good result.

Re:How the Claude Code team thinks about verification by fg137 on Tuesday November 17, 2026 @09:48AM (#50532133)
While I have seen numerous tickets addressed by Claude/Codex in one shot with decent solutions, about half of their first implementation is completely wrong in the direction, is way over engineered/significantly expands the scope, or leads to very confusing or even incorrect user-facing behavior.

I can't see how I would be able to keep customers happy without actually reading the code and understand what it is doing.

Re:How the Claude Code team thinks about verification by rolosa on Tuesday November 17, 2026 @12:11PM (#50534616)
The argument that engineers like the one in question are needed because of their judgment, guidance, experience, etc. But on the other hand if their job is reduced down to "@grok review and fix issue #42069", every 5 minutes of their workday, then what value do they actually add? Why not just remove them from the equation? I'm saying that a developer who brags about doing 2000+ prs/day is effectively dead weight and brings nothing to the table.
Re:How the Claude Code team thinks about verification by senordevnyc on Tuesday November 17, 2026 @08:29PM (#50540719)
Well, she’s using the new tools available to us to pioneer these kinds of scalable software factories and using that process to build a very successful product.

It sounds like you’re doing things basically the way they’ve always been done.

Re:How the Claude Code team thinks about verification by magicalist on Tuesday November 17, 2026 @10:45PM (#50541606)
> It sounds like you’re doing things...

What sounds like that exactly? I don't see anything in the GP about what and how they're "doing things" one way or the other.

They're making an argument about someone's statements. Engage with the argument or don't, but don't make it weird like this.

Re:How the Claude Code team thinks about verification by senordevnyc on Wednesday November 18, 2026 @07:45AM (#50545644)
This comment: https://news.ycombinator.com/item?id=50531344

This is so far from engaging with how Lauren actually works that’s it’s barely worth even responding to. They’re describing a standard software engineer workflow as it existed decades ago. Lauren is playing on a totally different level, more successfully I suspect (though who knows because while she’s public, this critic is anonymous).

Re:How the Claude Code team thinks about verification by losteric on Tuesday November 17, 2026 @10:57PM (#50541700)
Maybe the value is in setting up that system and the skillset to know when to stop/sidestep/backtrack that mechanism?

Basically management I guess

Re:How the Claude Code team thinks about verification by Alacart on Tuesday November 17, 2026 @10:13AM (#50532477)
While I suspect your point about the value/quality of the work at 2000 PRs/month is directionally correct, isn’t their entire take that they’re no longer micromanaging the PRs at all? So talking about how much time they commit to each one doesn’t make much sense. It’s a bit like looking at a modern car and saying “haha how will they even get the horses to pull this?” or maybe “how do these developers know the compiler isn’t using GOTO statements underneath?”.

Now, are those PRs doing useless work? Are they introducing unnecessary risks or trade offs? Are they building towards something impossible to understand or that will fall apart under its own sloppy weight?

Those seem more like the right questions to me, in the long run, and I suspect that’s what you really have in mind. Because if the answers are no, then why wouldn’t you be doing as many PRs as possible?

Re:How the Claude Code team thinks about verification by JambalayaJimbo on Tuesday November 17, 2026 @11:06AM (#50533380)
A PR to me represents a specific boundary of human ... review. It's a unit of work that is typically reviewed by at least 2 team members. If I'm working solo on some project, I don't use PRs.

So to me "not micromanaging PRs" doesn't make any sense at all. Bragging about how many PRs you can get merged is like bragging about how many WPM you can get when prompting an AI instead of manually typing.

Re:How the Claude Code team thinks about verification by ffsm8 on Tuesday November 17, 2026 @02:15PM (#50536676)
That's been the way I've worked in previous jobs too. Nonetheless some PRs need less review, others more. It's a gradient, and they're merely moving the range further towards "doesn't need in-depth" by introducing other quality gates.

That comes with trade-offs, but some may be willing to accept those. It's ultimately the decision of the developers and enterprise how they want to build their product, can't really say anything else about this

Re:How the Claude Code team thinks about verification by jwilliams on Tuesday November 17, 2026 @04:27PM (#50538438)
Are you arguing that the PR is unnecessary for AI/agents? Or that human review is mandatory?

All productivity metrics have flaws (usually deep ones). Doesn’t mean they’re not instructional though.

Re:How the Claude Code team thinks about verification by jimbokun on Tuesday November 17, 2026 @05:15PM (#50539024)
PRs are part of a software development process meant for collaborating humans. If no humans are reviewing anything, having PRs as part of the process is pretty pointless.
Re:How the Claude Code team thinks about verification by jwilliams on Tuesday November 17, 2026 @07:03PM (#50540051)
I mean, I agree conceptually. But the PR is just a useful unit of work.

My PRs get assessed for if a human is needed. I could probably do that at the branch level, but PRs just has all the metadata and CI/CD tooling etc etc.

Re:How the Claude Code team thinks about verification by senordevnyc on Tuesday November 17, 2026 @08:25PM (#50540693)
Exactly this. I have repos where agents can merge their own PRs, I literally never look at them. It’s still a useful primitive.
Re:How the Claude Code team thinks about verification by dmoy on Tuesday November 17, 2026 @05:35PM (#50539258)
> Or that human review is mandatory?

Depending on the domain, human review can be mandatory. (i.e. if you're a public company and the thing you're touching can materially affect the company finances)

Re:How the Claude Code team thinks about verification by wanderlust123 on Tuesday November 17, 2026 @08:49PM (#50540869)
Frontier LLMs can do reviews equal to if not better than the average dev once setup properly with guidance.
Re:How the Claude Code team thinks about verification by WeakSpeller on Tuesday November 17, 2026 @10:09PM (#50541377)
You need to work with better average devs then.
Re:How the Claude Code team thinks about verification by fg137 on Tuesday November 17, 2026 @08:03AM (#50530817)
The talk makes that sound something new, but every team I know has already adopted that before AI. I really don't get what value that talk provides -- I learned nothing after watching it.
Re:How the Claude Code team thinks about verification by xnorswap on Tuesday November 17, 2026 @08:09AM (#50530868)
You are fortunate to have always worked in highly effective teams and organisations.

This reminds me of when "The Joel Test" was first published and it was met by people unbelieving there was a world out there who weren't even using source control at the time, let alone a bug-tracker.

Re:How the Claude Code team thinks about verification by JambalayaJimbo on Tuesday November 17, 2026 @11:00AM (#50533264)
I have a friend who worked for a billion dollar company (a household name here in Canada) who conferred that they weren't using source control. As late as 2019.
Re:How the Claude Code team thinks about verification by vinnymac on Tuesday November 17, 2026 @02:03PM (#50536492)
Reminds me of a DevOps conference I attended in 2019 where everyone in the room hadn't heard of a feature flag before, and wanted me to share my experiences with them so they could better understand how to use them in their enterprise software. Meanwhile I had been using them since I was an intern, and was confused why all these graybeards were listening to me talk about anything from a db table to a full fledged company like LaunchDarkly, Optimizely, etc.
Re:How the Claude Code team thinks about verification by jimbokun on Tuesday November 17, 2026 @05:10PM (#50538965)
Good on those graybeards for wanting to learn new things.
Re:How the Claude Code team thinks about verification by solyin on Tuesday November 17, 2026 @06:38PM (#50539804)
We are our own bubbles of competence.
Re:How the Claude Code team thinks about verification by jamesfinlayson on Tuesday November 17, 2026 @11:28PM (#50541883)
I hope their beards really were grey - I work with a guy who worked somewhere in the mid 1990s and there were two servers (mainframes) - a development server and the production server. I could believe feature flags weren't a thing at that company, but we've also come a long way.
Re:How the Claude Code team thinks about verification by SatvikBeri on Tuesday November 17, 2026 @03:40PM (#50537860)
For some contrast, my first job where the team used source control was in 2015 – 15 years after the Joel test came out. That was after switching jobs in manufacturing or finance to tech.
Re:How the Claude Code team thinks about verification by stingraycharles on Tuesday November 17, 2026 @06:40PM (#50539829)
Believe it or not, there still are so many places that I’m working with that have hundreds of thousands of lines of code and no source control. They just pass zip files around and whatnot. There’s a whole parallel universe invisible to the average HN reader.
Re:How the Claude Code team thinks about verification by dominotw on Tuesday November 17, 2026 @09:34AM (#50531928)
ai engineer channel was good for a brief moment. Now its just slop.

Love to find some high quality youtubes to listen too to reduce my screen time.

Re:How the Claude Code team thinks about verification by ntcho on Tuesday November 17, 2026 @12:31PM (#50534982)
[delayed]
Re:How the Claude Code team thinks about verification by khasan222 on Tuesday November 17, 2026 @01:39PM (#50536117)
Absolutely teams have. But for some reason we expect the ai to write code better than us, so we don’t hold it to the same standards all the time.
Re:How the Claude Code team thinks about verification by verelo on Tuesday November 17, 2026 @10:19AM (#50532564)
I wish my reaction to seeing this post was "oh wow Anthropic never write stuff like this, I should make time to read it". My gut instinct was to say "this is going to be long, over complicated and not worth my time". I really think this is the influence of the 90's - 00's SEO. Write lots of stuff, hope Google cares...get more business.
Re:How the Claude Code team thinks about verification by Creamsicle47 on Tuesday November 17, 2026 @01:26PM (#50535911)
Lauren Tan is a professional larper