It is more of a bunch of different things for working out source code correctness glued together and the EVA code that is a static analyzer has limitations, but it can be used to write proofs of the correctness of C code, so it is basically a tool to investigate after I have investigated the others.
> Aren't analyzers today part of the build pipeline form the get go? Especially as C is known to be full of booby traps.
No, since developers generally cannot be bothered to do that. You are lucky if you get -Wall -Werror. I have been working on improving this in OpenZFS. Unfortunately, Coverity’s scan service is incompatible with PRs due to it not supporting branches and having weekly scan limits, but CodeQL was integrated a few months ago and I have plans to integrate Clang’s static analyzer. I am also looking into SonarCloud.
> Adding such tools later on in the development is like activating warnings post factum: You'll get drowned in issues.
So you understand my pain.
> Especially in such critical domains as file-systems I would actually expect that the developers are using "the best tools money can buy" (or at least the best OpenSource tools available).
I submitted a paper to Asia BSDCon 2023 explaining this. Assuming it is accepted, I will be giving a talk there. I would post the paper for others to read, but I still have 2 months before the final paper is due and I want to make some additional improvements to it before then. Also, they would still need to accept the paper; I will be informed of their decision next month.
> "Still fixing bugs found by some code analyzer" doesn't sound like someone should have much trust with their data in something like ZFS, to be honest… The statement sounds actually quite scary to me.
You should look at the coverity scan results for the Linux kernel. They are far worse.
https://scan.coverity.com/projects/linux
https://scan.coverity.com/projects/openzfs-zfs
At present, the ZFS Linux kernel module is at 0.14 unresolved defects per thousand lines. That is better than it’s in tree competitors:
* btrfs: 0.90
* ext*: 0.87
* jfs: 1.78
* reiserfs: 1.12
* xfs: 0.79
At the time of writing, the overall Linux kernel is at 0.48 while the entire OpenZFS tree is at 0.09, but sure, be worried about ZFS. Ignore Linux whose developers have a number of obvious unfixed bugs being reported by Coverity (and this does not even consider other static analyzers). I would send patches for some of them if mainline Linux did not have an annoying habit of ignoring many of the patches that I send. They have ignored enough that I have stopped sending patches since it is often a waste of time.
Anyway, static analysis has the benefit of checking little used execution paths, but it also has the downside of checking never used execution paths. It is often hard to reliably tell which are little used and which are never used. most of the static analysis fixes being done lately are fixing things not known to affect end users and the fixes are just being done for good measure.
That said, most of the remaining reports are not even real issues. I just have not yet gotten around to writing an explanation of why something is a false positive or rewriting code in which a non-issue was reported because I felt some minor revision would make the code better while silencing the static analyzer.
In addition, ZFS has plenty of other QA techniques that it employs, such as a test suite, stochastic testing and code review. All three are done on every pull request. I am not aware of another major filesystem that does that much QA. Let me know if you find one.
Also, you do have a point. My suggestion is that you go back to punched cards, and keep them in triplicate at different geographically distinct facilities under climate control. Ideally, those facilities will have armed security and be on different continents in places that are known to have low seismic activity. That should keep your data safer than it is on a single ZFS pool. ;)
I only found one actual bug. While I was happy to get that fixed, the overall result was disappointing.
My conclusion that the best path forward was to design a language where a static analysis tool would add no value.
The only static analyzers that I am allowed to say have stood out to me are:
* Clang's static analyzer
* GCC's static analyzer with -Wno-analyzer-malloc-leak -Wno-analyzer-use-of-uninitialized-value -Wno-analyzer-null-dereference because those three checks are uniquely buggy. `-Wanalyzer-null-dereference` fails to understand assertions. `-Wanalyzer-use-of-uninitialized-value` fails to recognize initialization from pass by pointer. `-Wanalyzer-malloc-leak` is so buggy that out of the 32 reports it made, only 1 was real, but it was right before an `exit()` call in a test suite, so it was not a particularly interesting report. Unfortunately, I do not have notes describing why that check is buggy.
Note that I am restricted by Coverity's Scan User Agreement from saying anything involving Coverity that could constitute a comparison. I suggest that you look at the public information about its use by myself and draw your own conclusions. Keep in mind, your conclusions are not things that I said.
I once told the Coverity staff that the purpose of D was to put Coverity out of business (!)
That was probably a bad choice. John Carmack did not rate it very highly:
http://www.sevangelatos.com/john-carmack-on-static-code-anal...
https://analysis-tools.dev/tag/cpp
The only one I know to exist absent from that list is Microsoft’s /analyze.
That said, I am surprised that D only has 1 tool listed:
https://learn.microsoft.com/en-us/cpp/build/reference/analyz...
The only reason I know about it is that John Carmack rated it highly:
http://www.sevangelatos.com/john-carmack-on-static-code-anal...
How does this work?
(This is all in functions annotated with `@safe`.)
Try it and see for yourself.
There's a long list of things like this D does. For another example, it won't allow reading from uninitialized variables. It will check your printf/scanf format string against the arguments. (This last one was a big win.)
Clang and GCC are able to do this on C/C++ code via the format function attribute:
https://gcc.gnu.org/onlinedocs/gcc/Common-Function-Attribute...
I agree about it being a big win. I have been dragging my feet on retrofitting the OpenZFS source code to use it since I have just not had much time to revise my pull request doing it after code review asked me to use `(u_longlong_t)` instead of PRId64 and friends and the initial attempt to retrofit the OpenZFS codebase using it did not find anything I considered to be a real bug. That might be because a number of real bugs in this area had already been identified and fixed thanks to static analyzers.
That said, there is this annoying oscillation that both GCC and Clang have where they complain that my type is long long and I should use %lld when I use %ld for uint64_t, only to complain that my type is long and I should use %ld when I use %lld for uint64_t. Using `PRId64` from stdint.h weirdly suppresses that behavior. Using a typecast to `(u_longlong_t)` also suppresses it.
Think of all the stupid problems arising in C from `long` not having a defined size. I can't believe all the time I've lost dealing with that. In D, a `long` is 64 bits. Period. Problems just vanish into the ether.
But why not for instance use a build system in some "container"?
In other language environments that's quite common. It's easy for developers, so there is no issue. I think the project could "bother" contributors with something like that, couldn't it?
> Coverity […] CodeQL […] SonarCloud
An embedded C developer I've talked with quite often on some other forum, who imho is quite competent, said that Coverity is a poor tool that generates way too much false negatives and overlooks at the same time glaring issues. He was not happy about it. Said that's mostly an issue with all OpenSource tools for static C analysis. OTOH the commercial ones are very expensive usually, with a target market of critical things like aviation of safety systems in cars and military use, places where they spend billions on projects. Nothing there for the average company, and especially not for (frankly often underfunded) OpenSource projects.
CodeQL? It's mostly an semantic search and replace tool, as I know? Is it that helpful? (I had a look, but the projects I'm working on don't require it. One would just use the IDE. No need for super large-scale refactorings, across projects, in our case).
SonarCloud, hmm… This one I've used (around web development though). But am not a fan of. It bundles other "scanner" tools, with varying quality and utility. At least what they had for the languages I've actively used it was mostly about "style issues". And when it showed real errors, the IDE would do the same… (The question then is how this could be committed in the first place. But OK, some people just don't care. For them you need additional checks like SonarCloud I guess.)
> Clang’s static analyzer
Heard good things about that one!
Wouldn't it be easy to add at least this to the build by using some "build container"?
> So you understand my pain.
Well, that's why I think something equivalent to `-Wall -Werror` should be switched on before writing the first line of code, in any language.
But as I just checked whether `-Werror` really does what I've expected I came across this here:
https://embeddedartistry.com/blog/2017/05/22/werror-is-not-y...
I have to admit that in the context of C/C++ development they have a point there.
One more reason for "Solution 2.: Capturing the build environment", I guess.
> I submitted a paper to Asia BSDCon 2023 explaining this.
Oh, cool! Good luck with that!
> You should look at the coverity scan results for the Linux kernel. They are far worse.
I've heard the rumors… (The C dude form above was also always talking about that).
But I didn't want to go into that as Linux code quality is a quite loaded topic sometimes.
> I am not aware of another major filesystem that does that much QA. Let me know if you find one.
OK, you have a point here.
I've never looked to close to be honest. Firstly, I have a very hard time reading C. Secondly, I don't want to get scared. And C code always scares me… (Likely because I'm mostly doing Scala, and especially the FP thing. So I get scared most of the time even by occasional `var`s)
> My suggestion is that you go back to punched cards, and keep them in triplicate at different geographically distinct facilities under climate control. Ideally, those facilities will have armed security and be on different continents in places that are known to have low seismic activity. That should keep your data safer than it is on a single ZFS pool.
Sure an option to consider.
But I guess I will stay with engraving my data into solid rock. Proven for at least hundred thousand years.
At least someone needs to preserve the cat pictures and meme of our current human era for the cockroach people of the distant future. I'm not sure they will have a compatible Linux kernel and compiler available to build the ZFS drivers, or even punch card readers…
I am not sure how this helps.
> I think the project could "bother" contributors with something like that, couldn't it?
Which project?
> An embedded C developer I've talked with quite often on some other forum, who imho is quite competent, said that Coverity is a poor tool that generates way too much false negatives and overlooks at the same time glaring issues.
He likely violated a license agreement with Coverity, since no one is allowed to say anything comparing Coverity to anything else.
> Said that's mostly an issue with all OpenSource tools for static C analysis.
I have been filing bug reports.
> OTOH the commercial ones are very expensive usually, with a target market of critical things like aviation of safety systems in cars and military use, places where they spend billions on projects. Nothing there for the average company, and especially not for (frankly often underfunded) OpenSource projects.
So you understand my pain.
> CodeQL? It's mostly an semantic search and replace tool, as I know? Is it that helpful? (I had a look, but the projects I'm working on don't require it. One would just use the IDE. No need for super large-scale refactorings, across projects, in our case).
I have never heard about this function. It is a static analyzer whose checks are written in the CodeQL language. However, it is very immature. When github acquired it, they banished the less reliable checks to the extended-and-security suite, leaving it only with about ~50 checks for C/C++ code. Those catch very little, although in the rare instances that they do catch things, the catches are somewhat amazing. Unfortunately, at least one of those checks provides technically correct, yet difficult to understand, explanations of the problem, so most developers would dismiss its reports as false positives despite it being correct:
https://github.com/github/codeql/issues/11744
There are probably more issues like that, but I have yet to see and report them.
> SonarCloud, hmm… This one I've used (around web development though). But am not a fan of. It bundles other "scanner" tools, with varying quality and utility. At least what they had for the languages I've actively used it was mostly about "style issues". And when it showed real errors, the IDE would do the same… (The question then is how this could be committed in the first place. But OK, some people just don't care. For them you need additional checks like SonarCloud I guess.)
It is supposed to be able to integrate into github's code scanning feature, so any newly detected issues are reported in the PR that generated them. Anyway, it is something that I am considering. I wanted to use it much sooner, but it required authorization to make changes to github on my behalf, which made me cautious about the manner in which I try it. It is basically at the bottom of my todo list right now.
> Wouldn't it be easy to add at least this to the build by using some "build container"?
I do not understand your question. To use it, we need a few things:
1. To be able to show any newly introduced defect reports in the PR that generated them shortly after it was filed.
2. To be able to scan the kernel modules since right now, it cannot due to a bad interaction between the build system and how compiler interposition is done. As of a few days ago, I have a bunch of hacks locally that enable kernel module scans, but this needs more work.
3. An easy way to mark false positives without doing commits.
Without those things in place, it is a dead on arrival proposition. OpenZFS already tried using CPPCheck without #1 and #3 in place and it was so painful that the project reversed course. That happened while I was on a multi-year sabbatical from the project, so I only know about it from seeing traces of it in the repository and asking others who were around for it about what happened.
> Well, that's why I think something equivalent to `-Wall -Werror` should be switched on before writing the first line of code, in any language.
OpenZFS has had that in place for more than a decade. I do not know precisely when it was first used (although I could look if anyone is particularly interested), but my guess is 2008 when ZFSOnLinux started. Perhaps it was done at Sun before then, but both events predate me. I became involved in 2012 and it is amazing to think that I am now considered one of the early OpenZFS contributors.
Interestingly, the earliest commits in the OpenZFS repository referencing static analysis are from 2009 (with the oldest commit being from 2008 when ZFSOnLinux started). Those commits are ports of changes from OpenSolaris based on defect reports made by Coverity. There would be no more commits mentioning static analysis until 2014 when I wrote patches fixing things reported by Clang's static analyzer. Coverity was (re)introduced in 2016.
As far as the current OpenZFS repository is concerned, knowledge of static analysis died with OpenSolaris and we lost an entire form of QA until we rediscovered it during attempts to improve QA years later.
That said, if you have suggestions on QA, I am willing to listen to them, although keep in mind that it takes time for me to do things, even if I think they are good ideas, and I already have a large number of ideas to implement since returning to the project earlier this year.
> But I guess I will stay with engraving my data into solid rock. Proven for at least hundred thousand years.
That method is no longer reliable due to acid rain. You would need to bury it in a tomb to protect it from acid rain. That has the pesky problem of the pointers being lost over time.
> At least someone needs to preserve the cat pictures and meme of our current human era for the cockroach people of the distant future. I'm not sure they will have a compatible Linux kernel and compiler available to build the ZFS drivers, or even punch card readers…
Github's code vault found a solution for that, although they used it on source code:
https://github.com/github/archive-program/blob/master/GUIDE....
I vaguely recall another effort trying to include the needed hardware in time capsules, but I could be misremembering.
> I am not sure how this helps.
> > I think the project could "bother" contributors with something like that, couldn't it?
> Which project?
OpenZFS. But this is kind of a misunderstanding on my side. I thought you were talking about convince for local developers. But you're talking about a CI system? Did I now understand correctly?
But still a local developer needs a local build environment that is similar to the one on the CI which will do the checks later on. It's a common practice to bundle the whole tool-chain and all kind of checkers and scanners in a "build container". You can run it than locally as on the CI.
But how to do bare metal tests on GitHub I actually don't know. Kernel divers are something special I guess. Not the usual workload that can be completely put into a container. One needs at least some VM setup. Never did that on GitHub.
> > An embedded C developer I've talked with quite often on some other forum, who imho is quite competent, said that Coverity is a poor tool that generates way too much false negatives and overlooks at the same time glaring issues.
> He likely violated a license agreement with Coverity, since no one is allowed to say anything comparing Coverity to anything else.
LOL
> I have never heard about this function. It is a static analyzer whose checks are written in the CodeQL language.
Need to take a second look I guess. I had it differently in mind, but OK.
Thanks for the pointer.
> That said, if you have suggestions on QA, I am willing to listen to them, although keep in mind that it takes time for me to do things, even if I think they are good ideas, and I already have a large number of ideas to implement since returning to the project earlier this year.
I'm not sure I can say much without having actually deeper insides into issues with the current QA setup.
It sounded like it would be difficult to introduce tools for static analysis. But I'm not completely sure whether this is more a technical issue or a developer convenience and workflow thingy.
At least some of such tools can be integrated into a fully packaged build tool-chain, like I said. This way you can rule out for example that someone commits stuff that triggers false positives in some code analysis tool that you can run also against your local build. Of course you still need checks on the CI in place. But those checks should actually never trigger by something that someone should have seen and resolved already locally (and if it does trigger this needs investigation usually anyway as the local build does not mirror the CI build in every aspect any more). This would relief the need to "ignore" checks on the CI without commits, I guess. Just do the checks you can do locally locally. Deliver the tools to do so in a convenient "package" ("container", VM image / config, whatever; you know, something like Vagrant).
Showing test results in PRs or even outright reject PRs based on test results can be done in GitHub with Actions. One can also trigger Actions by special comments, useful for example for checks and benchmarks that only need to run occasionally when something related changes. But I guess you know all that… :-)
Regarding code analysis: I think the stuff that Microsoft Research does is also interesting, and wasn't mentioned yet. But I'm not sure how practically relevant this is (especially in Unix land). They have at least on paper some tools for C code analysis. But no clue how stuff works in reality. Never heard directly from anybody who used this things.
I was actually asked by another OpenZFS developer about a good local tool for this. I did not have an answer then and I still do not have a great answer for it now, but I am getting closer to one. Some build system patches + Clang's static analyzer + codechecker seems to be the most promising combination since that would not only be able to cross translation unit analysis, but also be able to do differential reporting so only new reports are visible, but I still need to figure out all of the details.
Also, I suspect the end result will be far less useful in practice than it sounds because of just how long the analysis takes. We would need a way to do incremental analysis like we can do incremental builds for it to be truly useful, and I currently do not have an answer for incremental analysis, although I suspect maybe something could be done in CodeChecker based on filesystem time stamps.
Supposedly PVS Studio can do that, but as I explained my previous comment (in an edit that I probably made after you initially read it), PVS Studio is unlikely to be an option beyond its free trial period.
That said, QA would be better achieved by integrating these tools into our continuous integration infrastructure so that they could be run whenever a pull request is done so that all newly reported issues are made available to all reviewers and we can avoid merging things that introduce regressions that static analyzers can catch in the first place.
> But still a local developer needs a local build environment that is similar to the one on the CI which will do the checks later on. It's a common practice to bundle the whole tool-chain and all kind of checkers and scanners in a "build container". You can run it than locally as on the CI.
It is easy to do builds on any Linux / FreeBSD machine. The test suite really should be run in a VM since running experimental code on your kernel is not advisable and is not always feasible. Machines that have root on ZFS cannot have that code loaded into the running kernel without significant inconveniences, which include breaking the system if you mess up in a way that completely breaks the driver. You can still use the ZFS stochastic testing tool called ztest. That does testing on a userland version of ZFS, although it is different from running the test suite.
> But how to do bare metal tests on GitHub I actually don't know. Kernel divers are something special I guess. Not the usual workload that can be completely put into a container. One needs at least some VM setup. Never did that on GitHub.
I think we are talking about different things here. github can integrate with buildbots and has its own runners that you can use via github workflows, but running the test suite has nothing to do with static analyzers, which I thought were the focus of the discussion.
> Need to take a second look I guess. I had it differently in mind, but OK.
I revised my response to ask if you had mistaken CodeQL for Coccinelle.
> It sounded like it would be difficult to introduce tools for static analysis.
It is infrastructure work. It is doable, but it takes time. I have been working on it. The addition of CodeQL to github pull requests was the first thing to come out of that.
> But I'm not completely sure whether this is more a technical issue or a developer convenience and workflow thingy.
If the developer is inundated with reports, it is useless. That is why we need to ensure that developers only see new reports from their code changes. We also need to be able to mark existing reports as false positives so that we do not have a bunch of unresolved reports.
> At least some of such tools can be integrated into a fully packaged build tool-chain, like I said. This way you can rule out for example that someone commits stuff that triggers false positives in some code analysis tool that you can run also against your local build.
Commits can only be done by a select few people and they are only done following code review. Asking the reviewers to download the code locally to redundantly run all of these tools on their local machines (which also implies inundating them with reports from pre-existing issues) is a fantastic way to cause developer burn out. That might sound silly, but real problems occur when you require developers to do menial tasks that require them to process huge amounts of mostly useless information.
I have been through all of that myself and the burnout problem hit me. Excessive local testing is also a productivity killer, which becomes a morale killer when you put your tested patch in a pull request and then the buildbot not only finds a problem with it, but then demonstrates that all of your local testing was a waste of time since you now need to start from the beginning in retesting everything.
It is far better to have the tools' reports added to the pull request by a machine for the reviewers to see in the PR, with pre-existing issues filtered from those reports. It avoids the "please end my suffering now" part of the development process that occurs when developers manually run all of this locally every single time they make a small change.
> Of course you still need checks on the CI in place. But those checks should actually never trigger by something that someone should have seen and resolved already locally (and if it does trigger this needs investigation usually anyway as the local build does not mirror the CI build in every aspect any more).
They do all the time. You simply cannot test all of the architectures, OS version combinations, etcetera locally that are tested by the continuous integration infrastructure.
> This would relief the need to "ignore" checks on the CI without commits, I guess.
I do not understand.
> Just do the checks you can do locally locally.
I used to do this. My productivity was several times lower, I still missed things that the continuous integration software caught and I burned myself out doing menial work by needing to manually run all of this. I really do not recommend it.
Now I do much less of this and rely on the buildbot to do the menial work.
> Deliver the tools to do so in a convenient "package" ("container", VM image / config, whatever; you know, something like Vagrant).
I have a suspicion nobody would use it since everything they need is already fairly simple to setup.
> Showing test results in PRs or even outright reject PRs based on test results can be done in GitHub with Actions.
OpenZFS already does this.
> One can also trigger Actions by special comments, useful for example for checks and benchmarks that only need to run occasionally when something related changes. But I guess you know all that…
I did not know that part, but I find it to be more useful to always run all tests all the time.
> Regarding code analysis: I think the stuff that Microsoft Research does is also interesting, and wasn't mentioned yet. But I'm not sure how practically relevant this is (especially in Unix land). They have at least on paper some tools for C code analysis. But no clue how stuff works in reality. Never heard directly from anybody who used this things.
If they published them for third party use, I would be willing to look.
I use CodeChecker with ClangSA, CTU enabled and with Z3 for refutation. The code base is smaller so incremental checking is less a concern to me.
Still, we had a look. CC allows taking a list of files, and updating those files analysis only while keeping the existing results for other non listed files.
But unless something has changed this partial analysis just do what it's told, analyzing only the files given on the CLI. With CTU this may miss side effects: a modified file may impact other files using its function for example. It's possible to use CC own CTU info to derive these dependencies and extend the list of files.
Then there are modified header files, with the usual inclusion dependencies.
So if it's not provided "out of the box", it should be possible to have a layer on top taking a list of changed files, extending it with both CTU and header dependencies, and passing the extended list to CC for a safe update.
A product where the company is afraid of comparative reviews implies it just isn't a competitive product.
Feel free to compare D with any other programming language.
I don't have any experience with Coverity, but used another commercial static analyzer (SA) that I won't name in case there are similar legal limitations as with Coverity ;)
With the commercial tool I have the same problem you mention: too many false alarms. Most reports are a waste of time really. In the end, people tend to ignore the tool.
The best results I get with Clang SA, used through CodeChecker. Very few false alarms, and usually at least code smells. It's not sound of course, but is free and could spot a few nasty bugs. I recommend trying CC+ClangSA.
One important point is the ability to do cross files analysis. It is very common to use a value from a function in another file. With a "one file at a time" analysis no assumption can be made on this value range, so the analyzer must be conservative. This leads to a lot of false alarms in practice.
With cross file analysis (called "cross translation units" or CTU analysis in Clang SA), the SA can propagate constraints along paths traversing multiple files. This makes a huge difference in my experience. As I explained in another comment, this is well suited for an application: all paths come from `main`. For a library the result will depend on the test application(s) provided to the SA: the call to the library under test must cover all the function domains for the analysis not to miss some issue. Otherwise, it may use too restrictive constraints from the UT that are not relevant to real life use.
But it may not be enough: the commercial tool I use do make cross file analysis. It seems more limited than Clang SA though, and misses a lot.
Another thing that help IMHO is the use of Z3 as a checker. ClangSA uses a simple and fast range based analysis. As far as SA goes, this is a bit primitive and not as powerful as polyhedral analysis for example (which is more expensive).
But when Clang SA range analysis finds an issue, it is cross checked with the Z3 SMT solver, which can deal with more complex constraints, and possible rejected there. I do embedded development, and Z3 can understand bit fields and bit operations for example (trivial for a SMT solver with bit vectors). I never tried with and without using Z3 in this way, but I assume it helps.
In any case, I'm very happy with the results I get from ClangSA+CC with CTU and Z3 enabled (check your LLVM toolchain for Z3 support, OK in Debian stable for example). It's much better in practice than the costly commercial tool we still use, for now.
For cross compilation, there may be a few Clang/GCC differences to work-around. CodeChecker already handle some, but maybe not all. It's been manageable in our case. For Linux based development CC should work fine out of the box.
This looks interesting. It's based on abstract interpretation which is more or less the most powerful approach for imperative code available. (Because the way it works it's likely slow as hell though, I guess).
But it's closed source. One of this kind of products where you need to asks for the price… I think we all know what this means: It'll be laughably expensive.
I don't see any offer for OpenSource projects frankly.
> https://github.com/NASA-SW-VnV/ikos
Also abstract interpretation based. Looks less polished than the first one at first glance.
It's under some questionable license. According to OSI it's OpenSource. According to the FSF it's not. (The FSF argument sounds strong. They're right in my opinion. This NASA license does not look like OpenSource).
But an OpenSource project could use it for free I assume.
> https://github.com/static-analysis-engineering/CodeHawk-C
Much more constrained in scope than the other ones. But looks a little bit "too academic" imho: Uses its own C parser and such.
At least it's OpenSource under MIT license.
Thanks for the links either way! Good to know about some tools in case one would need them at some point.
> I have planned to try using them on OpenZFS for a while, but I am still busy reviewing and fixing reports made by conventional static analyzers.
Stupid question about usual C development practices (as I don't have much contact with that):
Aren't analyzers today part of the build pipeline form the get go? Especially as C is known to be full of booby traps.
Imho it shouldn't be even possible to push anything that has issues discovered by tools.
This should be the lowest barrier as most code analyzers are at most able to spot quite obvious problems (the commercial one above is likely an exception to this "rule"). When even the usual "stupid analyzer" sees issues than the code is very likely in a very bad shape.
Adding such tools later on in the development is like activating warnings post factum: You'll get drowned in issues.
Especially in such critical domains as file-systems I would actually expect that the developers are using "the best tools money can buy" (or at least the best OpenSource tools available).
"Still fixing bugs found by some code analyzer" doesn't sound like someone should have much trust with their data in something like ZFS, to be honest… The statement sounds actually quite scary to me.