Ah, brings to mind one of my favorite Dijsktra quotes, "Program testing can be used to show the presence of bugs, but never to show their absence!"

I've never understood that to mean that he wasn't in favor of automated testing, only that it's got its limits. In this case, they now know a test case that was missing.

Yup, in my previous gig we had an approach that if there is a bug in the code that the tests didn't catch, it's actually 2 bugs: a product and a test issue. Similarly, if a dev needed to have access to the live debugging session to diagnose the issue, it meant there was yet another bug (to improve debuggability of the product). It was quite nice, taught me a lot.

Indeed. This is why I find 100% unit test coverage can actually be harmful: it gives you an illusion of safety.

> What's special here is the bug is a memory corruption, and memory corruption bugs in such libraries are usually instantly security bugs.

Is that special? Are there buffer overflow bugs that are not security bugs? It could be just my bubble as a security consultant, since (to me) "buffer overflow" assumes remote code execution is a given. It's not my area of expertise, though, so perhaps indeed not all reachable buffer overflows are security issues. (Presuming no mitigations, of course, since those are separate from the bug itself.)

As a crude example, there sometimes are off-by-one bugs which allow the buffer to overflow by one byte, and that single overflowing byte is always 0 (the last bye in a zero-terminated string), and it overwrites data in a variable that doesn't affect anything meaningful, giving you a buffer overflow with no security impact.

Although single zero byte overflows are sometimes exploitable.

https://googleprojectzero.blogspot.com/2014/08/the-poisoned-...

> Are there buffer overflow bugs that are not security bugs? It could be just my bubble as a security consultant, since (to me) "buffer overflow" assumes remote code execution is a given.

Not necessarily:

1. Most compilers use aligned variables and fields unless forced not to with a flag. Going 3 bytes over an array of 12 bytes can result in an overflow that is never detectable at runtime because the extra memory being used is used exclusively by the bug.

2. Malloced memory is harder (but not impossible) to turn into an unintended execution because the pages may be marked for data only by the OS. The stack pages are ~marked~ [EDIT: ~NOT marked~] as executable.

There's probably millions of buffer overflows that will not only never be exploited, but will also never be detected.

Sorry, I mean the special part is "the bug itself is a memory corruption". The second sentence is a quick explanation for those not in our bubble.

Lots of bugs aren't particularly exploitable. As Tavis notes, one of the major problems here is that there's a structure containing function pointers adjacent to the buffer.

If all you have is a stack overflow you may have your work cut out for you.

Further, while this bug may exist, assertions elsewhere could have meant it was unexploitable. So in isolation it's an overflow but in context it's not reachable. That didn't happen here. This happens a lot actually.

Unit tests aren't really for bug catching, they're to ensure you haven't changed behavior when you don't expect to.

They enable refactoring code in ways not possible without them.

FWIW, this does not match my experience. I have caught lots of bugs with unit tests, especially in code that is fundamentally complex (because it does complex things, not because it needs polishing). OTOH, refactorings often span units because real simplification comes from changing the ways units interact, or even which units exist, so the tests have to be changed anyway.

Granted, even tests that have to be changed have some value in securing a refactoring.

If you are writing tests to check new code in tandem with writing that code (either via TDD, or some other code-test loop), or are writing tests for existing code, you can (and usually will) find and fix bugs. Likewise if you are investigating a problem and write one or more test cases to check the behaviour.

Once those tests have been written, then they act as regression tests like the parent comment notes.

On "unit tests", I view the behaviour of the class/function and all its dependencies as a single unit for the purpose of testing. I've never liked the idea of needing to mock out a class in order to test another class, just because the class being tested makes use of it. The only case where mocks/stubs/etc. should be used is when interacting with an external component like a database or HTTP API. -- You don't see people that do this mocking out a list, string or other library classes, so why should project classes be any different.

To clarify, when I wrote "the ways units interact", I was referring to units that represent external dependencies in some kind (often a database, as you said). Many refactorings change those interactions in some way.

I agree that there is no reason to mock data containers or other classes with fully self-contained behaviour.