News for hackers. Stuff that matters.
Re:How the Claude Code team thinks about verification by simianwords on Tuesday November 17, 2026 @07:18AM (#50530416)
I don't agree with "very expensive" claim. My friend works in a f500. They do releases like 2-3 times a week. Assuming each release uses $500 of tokens (higher estimate) just for verification, that's like $6000 a month and that's just a small percent over the tokens used for coding. What's the big deal? Their team brings in way more money than that - its a drop in the ocean.

I also know of companies that run very expensive test suites using spot instances in AWS that cost around $100 each time. No one complains about it.

I think all the posts complaining about expenses forget to compare the orders of magnitudes.

Re:How the Claude Code team thinks about verification by xnorswap on Tuesday November 17, 2026 @07:49AM (#50530702)
It's not the per-run costs I worry about, it's the costs of churn.

As soon as Oracle Tests fail, the LLM has three options:

1. Rewrite the test - This undoes the benefit of the test in the first place 2. Implement workarounds and quirks to match the oracle, often at the expense of recreating buggy behaviour 3. Implement both corrected behaviour and quirks-mode behaviour to meet the oracle, raise further jobs to formalise the switch over and change of behaviour.

( And this "change of behaviour" might be a few pixels out of place! Or it could be a much more significant change. When trying to meet a goal, LLMs will often treat these with the same zeal. )

If you're working interactively, you can catch it and steer it to what's best for the feature. If you're fully automating development, then you're at the mercy of whatever the model does at that moment.

Oracle-based testing has its place, but I don't think it's good for large scale automated testing.

Also, you're talking about $500/release 2-3 times a week. That's fine, that's less than the cost of an employee.

Anthropic and others are talking about how to scale to merge 2000-3000 PRs per week.

At that scale, that cost per merge is $6M/month, surely a cost you'd want to minimise if you can cut them by half with better type-checking, static analysis, and other cheap deterministic tests, even if that's still a small cost of the whole project.

If nothing else, it's pure waste. Our planet is (mostly metaphorically) burning, and I believe that more efficient IT processes and development practices would go a long way to helping mitigate some of that.

Re:How the Claude Code team thinks about verification by simianwords on Tuesday November 17, 2026 @12:06PM (#50534515)
It’s not a good sign that you had to bring up the environment to justify this..
Re:How the Claude Code team thinks about verification by nozzlegear on Tuesday November 17, 2026 @01:33PM (#50536027)
[delayed]
Re:How the Claude Code team thinks about verification by Rapzid on Tuesday November 17, 2026 @04:10PM (#50538232)
That's a weak man fallacy. The strongest parts of the argument are costs and efficiency.
Re:How the Claude Code team thinks about verification by simianwords on Tuesday November 17, 2026 @06:17PM (#50539599)
But the author also did a fallacy

https://kenrahn.com/JFK/Critical_thinking/What_do_many_piece...

If one’s original argument doesn’t have enough weight, they just add other weak ones to justify it.

Anyway, responding to the strongest version of it: the economics of any new system always looks bad before it becomes popular. I heard people say the same thing about AI driven coding and now there’s absolutely no going back. If you were a software engineer living in 1990s and you heard someone say that Uber, a ride sharing company, spends 100s of millions per year on just microservices and db infra, you’d be skeptical again.

Usually the costs justify themselves because of the productivity it brings. People spend 10x on cloud for this reason.

If you think I’m wrong, let’s make a bet of sorts. I’m highly confident that AI driven testing will be another normal and extensively used tool just like how CI/CD as a concept is today

Re:How the Claude Code team thinks about verification by cicko on Tuesday November 17, 2026 @08:03AM (#50530811)
Come back to Earth, please. Not everyone wants to suddenly spend $6K more per month on deployments.
Re:How the Claude Code team thinks about verification by tayo42 on Tuesday November 17, 2026 @10:37AM (#50532834)
People really work in different worlds.

Most jobs I worked in wouldn't bother discussing 6k of expenses. The meetings and decisions making involved probably cost more

Re:How the Claude Code team thinks about verification by theshrike79 on Tuesday November 17, 2026 @04:13PM (#50538272)
[delayed]