But a codebase with good test coverage allows you to safely perform large scale refactorings without having regressions and that's useful property even if you have bugs and the refactoring preserves them faithfully.
The risk of using a tool that generates tests designed to encode the current behaviour is that you may be lulled in a false sense of safety, while all you've done is to encode the current behaviour, as advertised.
Perhaps this problem can be just solved by not calling these tests "tests" but something like "behavioural snapshots" or something like that (cannot think of a better name, but the idea is to capture the idea that they were not meant to encode necessarily the correct behaviour but just the current behaviour)
aka snapshot tests.
Code coverage metrics are most easily met by writing unit tests. Unit tests are tedious to write.
If you have a robot writing unit tests, I do not want to see your codebase. Refactor all day long. I’m not going near it.
Those tests tend to have very low value with testing significant refactoring, as the code the tests covers often won't exist anymore.
This is an instance of a more general problem, which I call "the problem of unwanted change". If you have an automated system which can change itself, how do you know a change is actually intended/correct or merely a symptom of a bug, failure or imperfect knowledge the automation has?
That's why I think human supervision is always needed to an extent, to determine what scenario has occured.
This happens in all sorts of systems. And people tend to think they can solve this with just another layer of automation like here. Testing was originally invented as a way to check if the program works correctly. If you automate it, you will face the same problem, just with a bigger code (in the form of tests rather than assertions).
Of course these kinds of tests are no replacement for tests that check actual requirements.
The combination of pieces is what should be tested then, not the behavior of each individual piece “just in case something needs it that way”. You’ll never be able to change anything with that approach. And why shouldn’t bugs be fixed anyways?
but why couldn't this be done even with the llm generated test cases?
Might also be useful when you want to refactor a legacy system or try to figure out the input space for a module, library or method.
For a new project or a project under active development, I agree that auto generating tests is probably a bad idea. But there are countless legacy systems with low coverage that are in maintenance mode. For those, generating tests that verify the current behavior is super useful. It lets someone make a change, and see that everything else stayed the same.
Throw away mandatory code coverage tech debt instead of adding artificially-intelligent tech debt on top.
The best I can see a use for something like this is more like a linter than a test writer. “Robot, find weird things in the code and bring them to me for review.”
Then you write the tests yourself.
If LLMs can help you to scaffold a test, and make it easy for you to write the business logic validation, I think that would be a big win.
On the other hand, if the generated tests are like most UTs, they will be over-coupled to the implementation and therefore slow down development. You might even see folks start deleting all the tests and regenerating them as part of a big diff, if it’s too hard to fix individual test cases that are failing due to coupling rather than any logic issue.
The problem I have with LLM generated tests is that it seems highly likely that they'd "ratify" buggy behavior, and I'd think that'd be especially likely if the code-base already had low test coverage. One of the nice things about writing new tests by hand is you've got someone who can judge if it's the system being stupid or if it's the test.
At a minimum they should be segregated in a special test folder, so they can be treated with an appropriate level of suspicion.