People who work on legacy code bases often build what are called “characterisation tests” - tests which define how the current code base actually behaves, as opposed to how some human believes it ought to behave. They enable you to rewrite/refactor/rearchitect code while minimising the risk of introducing regressions. The problem with many legacy code bases is nobody understands how they are supposed to work, sometimes even the users believe it is supposed to work a certain way which is different from how it actually does - but the most important thing is to avoid changing behaviour except when changes are explicitly desired.

An LLM doesn’t have to always get it right to be useful-have it generate a whole bunch of tests, run them all, keep the ones which hit new lines/conditions, maybe even feed those results back in to see if it can iteratively improve, stop when it is no longer generating useful tests. Hopefully, that addresses most of the low-hanging fruit, and leaves the harder cases to a human.

There already exist automated test generation systems which can do some of this–for example, concolic testing-but an LLM can be viewed as just another tool in the toolbox, which may sometimes be able to generate tests which concolic testing can’t, or possibly produce the same tests quicker than concolic testing would. There is also the potential for them to interact synergistically - the LLM might produce a test which concolic testing couldn’t, but then concolic testing might then use that to discover further tests which the LLM couldn’t.

Characterisation tests are not supposed to be tightly coupled – they are supposed to be integration/end-to-end tests not unit tests – the point is to ensure that some business process continues to produce the same outputs given the same inputs, not that the internals of how it produces that output are unchanged. Code coverage is used as an (imperfect) measure of how complete your set of test inputs is, and as a tool to help discover new test inputs, and minimise test inputs (if two test inputs all hit the same lines/branches, maybe it is wasteful to keep both of them–although it isn't just about code coverage, e.g. extreme values such as maximums and minimums can be valuable in the test suite even if they don't actually increase coverage.)

They can take the form of unit tests if you are focusing on refactoring a specific component, and want to ensure its interactions with the rest of the application are not changed. But at some point, a larger redesign may get rid of that component entirely, at which point you can throw those unit tests away, but you'll likely keep the system-level tests

An issue with going with llms will be to validate if the behavior described are merely tolerated or if they're correct. Another will be wether something is actually tested (e.g. a code change still wouldn't break the test). Too granular output check would be an issue as well.

All in all this feels like a bad idea, but I hope to be wrong.

Characterization tests ideally need to not be written in code but defined in something resembling a configuration language - something without loops, conditionals, methods, etc. There then needs to be a strict separation of concerns kept between these definitions and code that executes them.

I wrote a testing framework (with the same name as my username) centered around this idea. Because it is YAML based, expected textual outputs can be automatically written into the test from actual inputs which saves tons of time and you can generate stakeholder documentation that validates behavior.

It might seem unbelievable but with decent abstractions, writing tests and TDD stops being a chore and actually starts being fun - something you won't want to delegate to an LLM.

The technical aspect of the code is not actually the difficult part WRT maintenance. Someone that knows COBOL as a language can figure out what some code does and how it does it. It takes time but that is information that can be derived if you just have the code.

The main problem with COBOL is the code is often an implementation of some business or regulatory process. The COBOL maintainer retiring is taking knowledge of the code but more importantly the knowledge of the literal business logic.

The business logic and accounting/legal restraints aren't something that can necessarily be derived from the code. You can know some bit of code multiplies a value to 100 but you can't necessarily know if it supposed to do that. If the source code and documentation don't capture the why of the code the how doesn't help the future maintainer.

Often with COBOL the people that originally defined the why of the code are not just retired but dead. The first generation of maintainers may only have ever received partial knowledge of the why so even the best documenters have holes in their knowledge. They may have had exposure to the original why defines but neglected or didn't have an opportunity to document some aspects of why. The subsequent generations of maintainers are constrained by how much of the original why was documented.

Edit: pre-coffee typo

I feel the same way about how test code is viewed even outside of AI. A lot of the time the test code is treated as a lower priority code given to more junior engineers, which seems like the opposite of what you would want.

Getting LLMs to write tests is like getting LLMs to write my spec.

For the remaining 95% of uninteresting surfaces I'd be perfectly happy to let an AI interpolate between my key cases and write the tests that I was mostly not going to bother writing anyway.

Most systems are pretty predictable. it("displays the user's name") isn't very novel, and is probably pretty easy for a LLM to generate.

Arguably the implementation that passes such a test is even simpler, making it a bit questionable why we have the human write that part.

It's nowhere as fancy as FB tool but I know it is blessed by company.

I’m guessing they want to automate tests because most engineers skimp on them. Compensating for lack of discipline.

This system would be a godsend in the minds of engineers who think / operate that way.

I've also had managers who told me I wasn't allowed to write tests firsts as it was slower. Luckily I was able to override / ignore them as I was on loan "take it up with my boss". They're probbably thinking the same as the above engineers.

Another way to think of this is most devs hate documentation... if they had an AI that would write great docs from the code they'd love it. And these to these devs docs they don't have to write are great docs :)

Sounds like a great place to work.

You're suppose to write them together, at the same time. What's slower is spending all of your time mousing around the GUI like a caveman and then having missing tests and then trying to debug every future recurring regression forever the rest of the project's lifespan.

The only thing that's slower is perhaps trying to do TDD for the first time. After that, you're embarrassed at how much time you wasted wandering in a browser before your API tests were finished.

And making those assertions shouldn't be mindless work. Documenting the failure cases should be the most interesting work of all the code you are writing. If you are finding that it isn't, then that tells you that you should be using a DSL that has already properly abstracted the failure cases for your problem space away.

* While "errors are values" makes it possible to enumerate the potential failure cases, most of the time errors are just strings. The thing you are forced to document by the lack of exception semantics is the sites at which you might be dealing with an error vs. a success value. In an IO heavy application this rounds up to "everywhere."

* Go is not really powered to do DSLs in a type-safe way, except through code generation. I would view the LLM as a type of code generator here.

* If your errors are strings, you are almost certainly doing something horribly wrong. Further, your tests would notice it is horribly wrong as making assertions on those strings would look pretty silly, so if your errors are strings that tells that your testing is horribly, horribly wrong.

* Go is not a DSL, no. It is unabashedly a systems language. Failure is the most interesting problem in systems. Again, if your failures aren’t interesting, you’re not building a system. You should not be using a systems language, you should be using a DSL.

When you have no idea what you are doing, choosing the wrong tool at every turn, an LLM might be able to help, sure.

I agree that the widespread use of code generation (LLM or otherwise) is revealing of this tension.

Do you mean Goa? Indeed, it is not a language fit for what we're talking about. It only allows description of services, not the implementation. For that you have to fall back to Go, which completely misses the whole reason for using a DSL.

> "one of your upstream calls might fail and then you return an error to your caller" is not that interesting.

Sure, and which is why you wouldn't use Go here. There is absolutely nothing in Go that is geared towards abstracting those kinds of things away. And you can't tell me that Goa tempts you. It is not a good DSL.

When you try to get the LLM to write the code, you find that it’s easier to get it to write the tests. So you do that and publish about that first.

Just in case any reader hasn’t tried this, the basic idea is to make statements about code’s behavior that are weaker than a totally closed-form proof system (which also have their place) stated as “properties” than are checked up to some inherently probabilistic bound, which can be quite useful statements.

The “canonical” example is reversing a string: two applications of string reverse is generally intended to produce the input. But with 1 line of code, you can check as many weird Unicode edge cases or whatever as you have time and electricity.

I know this example seems trite, but I met this because some hard CUDA hackers doing the autodiff and kernels and shit that became PyTorch used it to tremendous effect and probably got 5x the confidence in the code for half the effort/price.

It doesn’t always work out, but when it does it’s great, and LLMs seems to be able to get a Hypothesis case sort of, closer than starting from scratch.

In my experience, at least in languages like C++ or Java, unit tests are made of tedium, so I'm absolutely not surprised that the first instinct is to use LLMs to write that for you.

Sadly, this is a rare approach, so if you cooperate with others it's hard to use it.

I dread the morning after a night of getting something to work…somehow.

- Software engineering, which is akin to real engineering, as it involves desigining a complex mechanism that fits a lot of real world constraints. Usually you need to develop a sophisticated mental model and exploit it for the desired results. Involved implementations and algorithms usually fall into this category. - Talking to computers, which is about describing what you need to the computer. Usually focuses on the 'what' as the 'how' is trivial. Examples include HTML/CSS, Terraform and very simple programs (like porting a business process flow from a flowchart to code). And, indeed test code.

LLMs are terrible at the former, but great at the latter.

- TDD, which as you say describes the system's behavior. But it often deals with the nominal cases. It is hard to predict all that can go wrong in the initial development phase.

- tests designed to reproduce a bug. The goal of these to try very hard to make the system fail, taking inspiration with the bug's context

Maybe this LLM test generator could allow to be more proactive in the second kind?

AI-generated tests can work like compiler/sanitizer warnings. If they fail, you can audit them and decide if it was a true or false positive.