> it's not realistic to enforce unit test coverage % with a project at the scale of OpenSSL, right?

Why not?

You can enforce that all new files should be covered (at the very least line-covered). It requires some setup effort (collecting code coverage and either sending it to a tool which perform the correlation or correlating yourself), but once that's done... it does its thing.

Then you can work on increasing coverage for existing files, and ratcheting requirements.

It's realistic to reject a CR for a new parsing function without proof that it works, which usually comes in the form of a unit test

Nit: a unit test never proves that it works. At best it can prove that it doesn't work.

Otherwise, I agree.

Nit: "never"? Even if you unit test all cases?

Yes, never. You are assuming that the implementation of all unit test cases are themselves correct (that they would fail if there was any error in the case they cover). In fact unit tests are often wrong. In that context a unit test can't even prove code incorrect, unless we know that the unit test is correct.

IMO to prove that code is correct requires a proof; a unit test can only provide evidence suggestive of correctness.

Mistakes in proofs are just as probable as mistakes in exhaustive tests.

An exhaustive test is just one type of a machine verified proof.

> An exhaustive test is just one type of a machine verified proof.

Not entirely sure I agree with this. A proof by construction is a very different beast to empirical unit tests that only cover a subset of inputs. The equivalent would be units tests that cover every single possible input.

> The equivalent would be units tests that cover every single possible input.

That's what "exhaustive" means.

Touché

How do you know you covered all cases. You can verify you cover all cases the code handles easily enough. However does the code actually handle all the cases that could exist? A formal proof can bring to your attention a case that you didn't handle at all.

Formal proofs also have limits. Donald Knuth once famously wrote "beware of bugs in the above code, I proved it correct but never ran it". Which is why I think we should write tests for code as well as formally prove it. (On the later I've never figured out how to prove my code - writing C++ I'm not sure if it is possible but I'd like to)

There are limited cases where this is possible. Like a pure function that takes a single 32bit number as input. Just go over all the possible inputs.

You can't as a verify all the outputs though, only that it doesn't crash, which isn't useful for detecting edge cases where the results are wrong but don't crash. And that is a single pure function, if that pure function with a wrong output is then fed into/used by some other function (a non-pure data storage in particular - a large part of what computers are used for is storage of some form, not pure functions) you can read off the end of the buffer or other bad things.

Even running through all 4 billion some cases a single 32 bit number can result in your test taking a significant amount of time - enough that you wouldn't want to run it very often. One value of real world tests is often that they can detect that you broke what you thought was a completely unrelated area of code.

Seriously? It is pretty easy to imagine function which has limited and low combinations count

Enums, bools, etc.

>can result in your test taking a significant amount of time - enough that you wouldn't want to run it very often.

It is irrelevant in theoretical discussions like this

It's worth noting that coverage can be a deceptive metric sometimes.

You can have coverage on code that divides - it won't tell you if you ever divide by zero.

You can have coverage on code that follows a pointer - it won't tell you if you ever pass a bad pointer.

yeah but there isn't even a try here

Not saying it isn't worth an attempt, just that the real meaning shouldn't be lost.

It should be realistic, line coverage isn't really that hard. The hard thing is that high line coverage alone is usually not enough for numerical stuff...

No one is asking for that. But this is code that did one thing: punycode decoding, with millions of well documented test vectors. The code had zero dependencies on anything OpenSSL related. It is a very simple "text in, text out" problem, the most trivial thing to write unit tests for. At the same time, it's code that parses externally provided buffers and has to deal with things like unicode in C - there should be a massive red flashing warning light in every developers head here.

It is trivial to enforce that new functions have new unit tests and fuzz tests. You are the reviewer of https://github.com/openssl/openssl/pull/9654 and you just say "Please add unit tests and fuzz tests for foo and bar" and you don't approve it.

I don't know what the deal is with their testing culture but in year 27 of the project they demonstrably haven't learned this lesson. It's nice that they added integration tests (testing given encoded certs) but as the article points out that was insufficient.

IMO, one of the biggest benefits of "modern" systems languages like Rust, D, Zig is how much easier they make writing and running tests compared to C and C++. Yes, you can write tests for those languages, but it's nowhere near as trivial. And that makes a difference.

I was writing unit tests for C in 1996, naturally we still haven't coined the term, so we just called them automatic tests.

It was part of our data structures and algorithms project, failure to execute the automatic tests meant no admission to the final exam.

We had three sets of tests, those provided initially at the begin of the term, those that we were expected to write ourselves, and a surprise set on the integration week at the end of the semester.

I am not buying it.

Writing unit tests for c/c++ is trivial. There are perfectly fine test frameworks, used by developers every day, integrated in any major IDE or runnable as one-liner from the command line.

This is absolutely a cultural problem.

In modern languages you either have built in testing or it’s a single package install away. It is nowhere near this simple with C or C++

Last week I code a "please add unit tests for this code". The person who wrote that comment wasn't aware that this was a refactoring where the functionality was well tested.

There is no substitute for reviewers who really understand the code in question. The problem is they are the ones writing the code and so are biased and not able to give a good review.