This is called a "golden master" (giving X input to the system and recording the output as a test expectation). The difference with the parent is that it is way less granular, so both have value.

This is something I've yet to see a software testing framework do. Compare the results of two different implementations separated in time. i.e. two different revisions out of source control, using one as the golden reference for the other.

bazhenov/tango does something like this for performance tests, basically to counter system behaviour you run the old and new implementation at the same time.

Isn’t that kind of what snapshot tests can do?

I maintained a fairly successful legacy C/C++ application that performed business critical functions. The core developers were long gone, and countless of changes had to gone into it since. The only way the application survived was automatically comparing outputs and intermediate data structures against "ideal" values. Hundreds of business scenarios with their own input datasets. No one really understood what each scenario or ideal value meant. So your goal was to minimize divergence, and update the test case if it was deemed acceptable.

The problem is xUnit style tests are really bad at this and make it tedious - which is why people gravitate to LLMs for writing them. LLMs on the surface look like they can relieve the pain of using bad abstractions - but theyre still a band aid on a gaping wound. 20 years ago we'd be using them to write ugly raw PHP.

Characterization tests ideally need to not be written in code but defined in something resembling a configuration language - something without loops, conditionals, methods, etc. There then needs to be a strict separation of concerns kept between these definitions and code that executes them.

I wrote a testing framework (with the same name as my username) centered around this idea. Because it is YAML based, expected textual outputs can be automatically written into the test from actual inputs which saves tons of time and you can generate stakeholder documentation that validates behavior.

It might seem unbelievable but with decent abstractions, writing tests and TDD stops being a chore and actually starts being fun - something you won't want to delegate to an LLM.