News for hackers. Stuff that matters.
Re:Property-based testing in practice [pdf] by IshKebab on Tuesday April 02, 2024 @03:52AM (#39903299)
I've never seen anyone do mutation testing with software (it's pretty common for hardware though). Does it require language level support?
Re:Property-based testing in practice [pdf] by boxed on Tuesday April 02, 2024 @04:02AM (#39903359)
I'm the author of mutmut, the primary mutation tester for python, so I think I can speak a bit on this.

It's quite straight forward to do MT in software. I've done quite a bit of it for specific libraries that I've built (iommi, tri.struct, and internal code). A big advantage in my book of MT over PBT is the much lower cognitive overhead for MT and that you can know you have done it completely. The second may be just a false sense of security or an emotional blanket, but still.

I have written about mutation testing a few times: https://kodare.net/2019/04/10/mutation-vs-property-based-tes... https://kodare.net/2018/11/18/the-missing-mutant-a-performan... https://kodare.net/2016/12/12/mutation-testing-in-practice.h... and my talk on mutmut from pycon sweden is on youtube: https://www.youtube.com/watch?v=fZwB1gQBwnU

Re:Property-based testing in practice [pdf] by IshKebab on Tuesday April 02, 2024 @08:02AM (#39904772)
Does that only work in Python though? What about compiled languages? You don't really want to have to recompile your whole project again for every line you change...
Re:Property-based testing in practice [pdf] by boxed on Tuesday April 02, 2024 @09:34AM (#39905586)
mutmut right now is implemented by writing changes to disk and starting new processes. But you can implement it via "mutation schemata" where you functionally compile all possible mutants ahead of time, plus the original function, and replace the original function with a trampoline that either calls the original or one of the mutants depending on some external state.

I have a prototype of mutmut that does this and it's 10 to 100x faster. It does have the downside of not being able to mutate stuff like static global variables and such though.

Re:Property-based testing in practice [pdf] by Smaug123 on Tuesday April 02, 2024 @04:04AM (#39903368)
[I've never used any mutation testing tools either.]

In general it doesn't require language-level support, of course - you can just make a change and rebuild it, à la Stryker https://stryker-mutator.io/docs/stryker-net/technical-refere... . PITest operates on JVM bytecode (https://pitest.org/) for orders of magnitude speedup.

Re:Property-based testing in practice [pdf] by IshKebab on Tuesday April 02, 2024 @02:14PM (#39909008)
Yeah I meant language level support to make it viable, which rebuilding a gazillion times doesn't really sound like.