News for hackers. Stuff that matters.
Re:Meta's new LLM-based test generator is a sneak peek to th... by wickedsickeune on Saturday February 24, 2024 @05:27AM (#39490553)
This is called a "golden master" (giving X input to the system and recording the output as a test expectation). The difference with the parent is that it is way less granular, so both have value.
Re:Meta's new LLM-based test generator is a sneak peek to th... by weebull on Saturday February 24, 2024 @07:23AM (#39490995)
This is something I've yet to see a software testing framework do. Compare the results of two different implementations separated in time. i.e. two different revisions out of source control, using one as the golden reference for the other.
Re:Meta's new LLM-based test generator is a sneak peek to th... by CHY872 on Saturday February 24, 2024 @08:31AM (#39491348)
bazhenov/tango does something like this for performance tests, basically to counter system behaviour you run the old and new implementation at the same time.
Re:Meta's new LLM-based test generator is a sneak peek to th... by adregan on Saturday February 24, 2024 @01:34PM (#39493944)
Isn’t that kind of what snapshot tests can do?
Re:Meta's new LLM-based test generator is a sneak peek to th... by fosstrack on Monday February 26, 2024 @01:21AM (#39508148)
I maintained a fairly successful legacy C/C++ application that performed business critical functions. The core developers were long gone, and countless of changes had to gone into it since. The only way the application survived was automatically comparing outputs and intermediate data structures against "ideal" values. Hundreds of business scenarios with their own input datasets. No one really understood what each scenario or ideal value meant. So your goal was to minimize divergence, and update the test case if it was deemed acceptable.