I understand. But that is a pretty silly comparison for him to make. He's sampling two different universes. In the open source universe, everything is open to him, by definition. He has a complete sample space. In the closed source universe, he may only sample the companies who choose to contract his services. That is an extremely biased sample.

Not to mention the basic fact that he's almost certainly not controlling for other factors, like project age, usage level, etc.

> CSS doesn't mean no one can see the code or that there aren't third-party audits.

It does mean that there are no meaningful third-party audits. An audit (or benchmark, for that matter) that aims to be approved by the vendor is as useful as a rubber stamp.

This is sometimes true, but not always.

There are many auditors who will do a good and fair job.

This is especially true if the audit is not intended to be publicly shared, but rather for the internal consumption of the company in question.

I'd still say an audit where the reviewers are skilled professionals, but biased by the source of income, is better than the average open source project where few people will do such a thorough audit ever; likely none after the original author, excepting special cases like Linux.