News for hackers. Stuff that matters.
Re:Probabilistic Flakiness: How do you test your tests? by dragandj on Monday December 21, 2020 @08:37PM (#25501903)
Hmm. One has to wonder how do one test the test of your tests, then. And those tests of the tests of my tests too...
Re:Probabilistic Flakiness: How do you test your tests? by slaymaker1907 on Monday December 21, 2020 @10:13PM (#25502453)
The final test is always human review. The point of testing is to simplify the analysis of correctness for humans.

Also, ideally your code and test are somewhat independently written, whether it is whitebox or blackbox. If that is the case,you greatly reduce the chance that both the code and the test are wrong in such a way that the test still passes.

This is why writing unit tests for stuff like getters and setters is really dumb. It doesn't really simplify checking whether the code is correct unless you use something like meanbean and thus provides some degree of separation between the test and the implementation.

Re:Probabilistic Flakiness: How do you test your tests? by pfdietz on Monday December 21, 2020 @09:41PM (#25502282)
Mutation testing is a way to test your tests.
Re:Probabilistic Flakiness: How do you test your tests? by jaggederest on Monday December 21, 2020 @10:06PM (#25502410)
And will very quickly humble even the most diligent of testers :) For example, you're testing that this value is in a range - but are you testing both endpoints and whether they're inclusive or exclusive? In faster languages I enjoy quickcheck style testing which also reveals these things pretty quickly.
Re:Probabilistic Flakiness: How do you test your tests? by vlovich123 on Tuesday December 22, 2020 @12:18AM (#25503145)
Do not confuse end-to-end functional tests that this article is describing for smaller “unit” tests that you typically might express in quick check/mutation testing (at least as I understand the space).
Re:Probabilistic Flakiness: How do you test your tests? by mitchellhorning on Monday December 21, 2020 @08:49PM (#25501977)
tests of test tests are test tests, so just use your test testing tests to test your test tests
Re:Probabilistic Flakiness: How do you test your tests? by postalrat on Monday December 21, 2020 @10:13PM (#25502447)
Somehow you build a feedback loop of tests.