News for hackers. Stuff that matters.
Re:Controversial Programming Opinions (2012) by pjc50 on Friday December 04, 2020 @11:44AM (#25304437)
> most software tests should be integration tests that use randomized inputs.

Good for fuzzing, bad for determining whether the output is what you expected it to be, because finding out what the output should be is .. the reason you wrote the software.

I've seen this done with an automated spec translation tool that produced an unoptimised but correct H264 decoder and set of reference inputs to achieve coverage of the actual target decoder.

Re:Controversial Programming Opinions (2012) by loup-vaillant on Friday December 04, 2020 @12:45PM (#25305387)
> Good for fuzzing, bad for determining whether the output is what you expected it to be, because finding out what the output should be is .. the reason you wrote the software.

The solution for this is property based tests. Outputs almost never have maximum entropy. They have invariants, and they're related to the inputs in various ways.

Trivial example: authenticated encryption. The input is some random text, some random key, and some random nonce. The result is a message. One property to check is the fact that the message can successfully be decrypted with the same key, and the resulting plaintext will be the same as the random text from the beginning. Another property is that any message that is not the message you computed will fail to decrypt (assuming you pretend not to know the key when as you corrupt the message). Testing (crudely) that you can detect forgeries (or at least accidental corruptions).