An LLM doesn’t have to always get it right to be useful-have it generate a whole bunch of tests, run them all, keep the ones which hit new lines/conditions, maybe even feed those results back in to see if it can iteratively improve, stop when it is no longer generating useful tests. Hopefully, that addresses most of the low-hanging fruit, and leaves the harder cases to a human.
There already exist automated test generation systems which can do some of this–for example, concolic testing-but an LLM can be viewed as just another tool in the toolbox, which may sometimes be able to generate tests which concolic testing can’t, or possibly produce the same tests quicker than concolic testing would. There is also the potential for them to interact synergistically - the LLM might produce a test which concolic testing couldn’t, but then concolic testing might then use that to discover further tests which the LLM couldn’t.
Characterisation tests are not supposed to be tightly coupled – they are supposed to be integration/end-to-end tests not unit tests – the point is to ensure that some business process continues to produce the same outputs given the same inputs, not that the internals of how it produces that output are unchanged. Code coverage is used as an (imperfect) measure of how complete your set of test inputs is, and as a tool to help discover new test inputs, and minimise test inputs (if two test inputs all hit the same lines/branches, maybe it is wasteful to keep both of them–although it isn't just about code coverage, e.g. extreme values such as maximums and minimums can be valuable in the test suite even if they don't actually increase coverage.)
They can take the form of unit tests if you are focusing on refactoring a specific component, and want to ensure its interactions with the rest of the application are not changed. But at some point, a larger redesign may get rid of that component entirely, at which point you can throw those unit tests away, but you'll likely keep the system-level tests
An issue with going with llms will be to validate if the behavior described are merely tolerated or if they're correct. Another will be wether something is actually tested (e.g. a code change still wouldn't break the test). Too granular output check would be an issue as well.
All in all this feels like a bad idea, but I hope to be wrong.
Characterization tests ideally need to not be written in code but defined in something resembling a configuration language - something without loops, conditionals, methods, etc. There then needs to be a strict separation of concerns kept between these definitions and code that executes them.
I wrote a testing framework (with the same name as my username) centered around this idea. Because it is YAML based, expected textual outputs can be automatically written into the test from actual inputs which saves tons of time and you can generate stakeholder documentation that validates behavior.
It might seem unbelievable but with decent abstractions, writing tests and TDD stops being a chore and actually starts being fun - something you won't want to delegate to an LLM.
The technical aspect of the code is not actually the difficult part WRT maintenance. Someone that knows COBOL as a language can figure out what some code does and how it does it. It takes time but that is information that can be derived if you just have the code.
The main problem with COBOL is the code is often an implementation of some business or regulatory process. The COBOL maintainer retiring is taking knowledge of the code but more importantly the knowledge of the literal business logic.
The business logic and accounting/legal restraints aren't something that can necessarily be derived from the code. You can know some bit of code multiplies a value to 100 but you can't necessarily know if it supposed to do that. If the source code and documentation don't capture the why of the code the how doesn't help the future maintainer.
Often with COBOL the people that originally defined the why of the code are not just retired but dead. The first generation of maintainers may only have ever received partial knowledge of the why so even the best documenters have holes in their knowledge. They may have had exposure to the original why defines but neglected or didn't have an opportunity to document some aspects of why. The subsequent generations of maintainers are constrained by how much of the original why was documented.
Edit: pre-coffee typo
People who work on legacy code bases often build what are called “characterisation tests” - tests which define how the current code base actually behaves, as opposed to how some human believes it ought to behave. They enable you to rewrite/refactor/rearchitect code while minimising the risk of introducing regressions. The problem with many legacy code bases is nobody understands how they are supposed to work, sometimes even the users believe it is supposed to work a certain way which is different from how it actually does - but the most important thing is to avoid changing behaviour except when changes are explicitly desired.