Re:How to avoid brittle code
by prodigal_erik on Monday March 28, 2016 @02:53PM
(#11376416)
Unit tests can verify assumptions that the implementation is internally relying on. E.g., say a greedy algorithm relies on a collection being sorted, even though the public interface does not return results in any particular order. If I make a mistake such that an edge case leaves the collection not sorted correctly, a test of the public API can't tell unless it happens to trigger the edge case whose importance might not be obvious.
> a test of the public API can't tell unless ...
"Can't tell unless" means exactly the same thing as "can tell". You have a way to get to that corner case through the public functionality.
If there really is no way to tell through the current public API, there the bug doesn't exist.
Re:How to avoid brittle code
by prodigal_erik on Monday March 28, 2016 @04:46PM
(#11377259)
I think I'm arguing for letting white-box unit tests make assertions about implementation details in a simple and clear way, rather than having to somehow devise a test using only the black-box API that happens to fail if the assertion doesn't hold. In other words, you can only guarantee parts of the internal state are valid if you know how to infer them from outside.
Are you arguing that all unit tests should be black-box?
You only need that sort of test in a programming language or public library, because you're publishing the behavior to unknown numbers of users, and it has to work, as documented. If they find too many things that don't work according to the documentation, they will correctly suspect that you aren't testing, and lose confidence.
But if some API is internal to an application, and no code depends on some aspect of its interface contract, then that aspect effectively doesn't exist.
It may happen that someone discovers this aspect in the documentation and tries using it. At that time it is either found broken, or found working. If it is found broken, it can be fixed. Or else it can be striken from the documentation (and another way can be found in the application code to meet its requirements).
If the aspect is found correct, or fixed to reflect the documentation, and is then used in the application, then that use constitutes a test. The feature is tested indirectly, through some regression test case which targets something in the application, which depends on that aspect of the API contract lower down.
If you design units with all kinds of detailed aspects in their interface contracts which end up not being used at all, and test all of them, it can be argued that you're wasting time: you're basically working on your library design hobby, at the expense of the project, which just serves as a vehicle for it.