Re:Testable IO in Haskell
by peteretep on Sunday June 21, 2015 @08:52PM
(#9755640)
> An example of "innocent intermittence" recently came up
> at my current employer: there was a test that invoked a
> web service twice, verifying idempotence. It then
> asserted the responses were identical.
This is not innocent intermittence, this is a bad test. No-one should get in trouble for it, but I would certainly expect senior developers to get involved to find out:
Why is idempotence being tested for on the response side rather than on the execution end? If the call is truly idempotent, then a changing date-stamp means that the test has captured the information envelope as well as the data payload, and is testing that, which is sloppy - you'd probably be upset to see the HTTP headers being tested in the same string, so why other pieces of the envelope too? If a test is calling a "web service", then it's either an integration test (so why's it testing for idempotence, rather than testing the envelope?) or it's a unit test gone awry. Are there other tests that do this too? Could this all be abstracted out in to a sensible library to make it easier for more junior developers to not make these mistakes?
The moment you start accepting less than 100% test pass rates as being OK, the moment madness descends and you lose much of the value of your test suite.
Why is idempotence being tested for on the response side rather than on the execution end? If the call is truly idempotent, then a changing date-stamp means that the test has captured the information envelope as well as the data payload, and is testing that, which is sloppy - you'd probably be upset to see the HTTP headers being tested in the same string, so why other pieces of the envelope too? If a test is calling a "web service", then it's either an integration test (so why's it testing for idempotence, rather than testing the envelope?) or it's a unit test gone awry. Are there other tests that do this too? Could this all be abstracted out in to a sensible library to make it easier for more junior developers to not make these mistakes?
The moment you start accepting less than 100% test pass rates as being OK, the moment madness descends and you lose much of the value of your test suite.