In particular constructing ( and often deconstructing on change ) differential tests is very expensive relative to cheaper things to set up.
I felt this article's argument was poorly presented because of this seeming contradiction. The video I linked was much clearer about what works and what doesn't when scaling to inhuman levels of PR merges.
I also know of companies that run very expensive test suites using spot instances in AWS that cost around $100 each time. No one complains about it.
I think all the posts complaining about expenses forget to compare the orders of magnitudes.
As soon as Oracle Tests fail, the LLM has three options:
1. Rewrite the test - This undoes the benefit of the test in the first place 2. Implement workarounds and quirks to match the oracle, often at the expense of recreating buggy behaviour 3. Implement both corrected behaviour and quirks-mode behaviour to meet the oracle, raise further jobs to formalise the switch over and change of behaviour.
( And this "change of behaviour" might be a few pixels out of place! Or it could be a much more significant change. When trying to meet a goal, LLMs will often treat these with the same zeal. )
If you're working interactively, you can catch it and steer it to what's best for the feature. If you're fully automating development, then you're at the mercy of whatever the model does at that moment.
Oracle-based testing has its place, but I don't think it's good for large scale automated testing.
Also, you're talking about $500/release 2-3 times a week. That's fine, that's less than the cost of an employee.
Anthropic and others are talking about how to scale to merge 2000-3000 PRs per week.
At that scale, that cost per merge is $6M/month, surely a cost you'd want to minimise if you can cut them by half with better type-checking, static analysis, and other cheap deterministic tests, even if that's still a small cost of the whole project.
If nothing else, it's pure waste. Our planet is (mostly metaphorically) burning, and I believe that more efficient IT processes and development practices would go a long way to helping mitigate some of that.
https://kenrahn.com/JFK/Critical_thinking/What_do_many_piece...
If one’s original argument doesn’t have enough weight, they just add other weak ones to justify it.
Anyway, responding to the strongest version of it: the economics of any new system always looks bad before it becomes popular. I heard people say the same thing about AI driven coding and now there’s absolutely no going back. If you were a software engineer living in 1990s and you heard someone say that Uber, a ride sharing company, spends 100s of millions per year on just microservices and db infra, you’d be skeptical again.
Usually the costs justify themselves because of the productivity it brings. People spend 10x on cloud for this reason.
If you think I’m wrong, let’s make a bet of sorts. I’m highly confident that AI driven testing will be another normal and extensively used tool just like how CI/CD as a concept is today
Most jobs I worked in wouldn't bother discussing 6k of expenses. The meetings and decisions making involved probably cost more
Sure you can have linting, compilation, unit tests, e2e tests. Do you really think Anthropic doesn't do this?