We have some customers that clearly come to us to get reports they can share with investors, and not that many come back so either they weren't happy or (I like to think) they've checked that box and aren't doing further tests. But never did one of them ask us to rephrase something for overt marketing reasons or provide even a single sentence of marketing material to include. Perhaps we were clear that this is not something we do and that's why they don't come back, though since they also don't ask and still order a test, that doesn't seem logical.
I also can't say that I've seen a Cure53 report where this is the case, including this one. If anything, the statement about a critical vulnerability in the previous pentest could make one weary, so I applaud Mozilla for publishing this rather than silent fixing and keeping it under wraps.
> Vendors pay for public-facing reports, as an extra line item in the [statement of work]
This is the first I hear of that, though the public scrutiny does mean we spend quite a bit of extra review time on such reports (from spell checking to going over every substantial statement and rating) so I could understand if Cure53 has different practices from ours. That does not mean they were bought (assuming, for the sake of argument, that a premium was paid, which I don't think there was).
> that norm should be that public-facing reports can be provided only in the same dry, technical form they're presented to development teams in ordinary, non-public projects
That is exactly what we do, and I am quite sure Cure53 works the same way in this regard. We review more deeply if we know ahead of time it's going to be public, as I wrote above, to make sure things are correct (especially since English is none of our native languages), but we don't alter the contents.
> I think third-party assessment reports like these are a real problem in our industry.
I really don't understand where this comment is coming from unless you have very different experiences with customers or public reports where you are based. At a minimum it doesn't apply to the report at hand. Too often these things are kept secret and it's a rare opportunity that we don't have to swear silence on our work. I'd love to do that more.
Audit reports like SOX and SSAE 16 that tie back to financial risk can be more staid, because of the legal risk to both the auditor and auditee. Even though it appears to be tied to financial audit, we could still accomplish a lot on the infosec side because of the close relation between infosec failure and significant financial impact. Final reports contained a section specifically for the company's own statements, and a section for the auditor statements that was almost always boilerplate straight from legal. Flourish and marketing were nowhere to be found.
As a former auditor I enjoy reading these reports but I always find myself asking: who are these "auditors" and how are they held accountable, besides not being invited back by the auditee? And: Have I ever seen such a report that concluded the product contained critical exceptions or findings that should discourage consumers from using the product? I can't think of any, maybe someone else can.
The problem is the ultimate consumer[1] of these reports, legal and procurement agents at buying companies, themselves don't care about the actual quality of the report except insofar as it satisfies their own transitive legal/sales requirements, and it's turtles all the way down. This harms users/customers at the end of the day because they don't have time to scrutinize the details of each individual report or have any real power. If we care about the end goal (secure, accessible software), we need for the auditing firms to collaborate we the government, judiciary, and ancillary vendors to tighten standards to include random[2], uniform checks (same auditor, same methodology, multiple vendors at once).
In the US, OSHA designates NRTLs like UL to perform safety testing, which are required everywhere from workplace standards to insurance requirements. In comparison, at least for accessibility, merely having your vendor have any assessment report is likely enough CYA to withstand a legal challenge. I acknowledge the power of recent website lawsuits to use the broader ADA to raise the bar here, but ADA's "enforcement through private lawsuits" enforcement mechanism is spotty and I think won't result in enough structural improvement.
[1] - These reports also serve as PR/marketing, which is probably moreso the case with Mozilla VPN, but in most enterprise software where these assessments are taking place, the marketing side is very much a secondary goal compared to the individual legal/sales relationship that hinges on the report.
[2] - I think removing the opportunity for vendors to fine-tune scope or prepare or respond to concerns (at least until the next review cycle) is a big step in the right direction, but unfortunately, the legal climate is very much all-or-nothing and not good at nuance. Section 508 (I'm not personally familiar with PCI/SOX/etc. but suspect those are similar) is formally speaking "all or nothing" check all the boxes things, and in that climate, good random audits will basically be always-failing, and if you make too hard a standard that even reasonable vendors can't meet with an earnest effort, you'll end up constricting the market into a meta-game of who can hack the auditing process. See federal government procurement.
I think you mean "later unpublished engagements".
⃰ And that it's even remotely readable to be an asterisk, which my terrible vision can't make out in the text input field on HN, and that it's wrapped correctly (which it's not), and that it's spaced far enough away from the previous character (which it's not) and that <SPACE> <ZWJ> <COMBINING ASTERISK ABOVE> are handled properly by the website (which is still not universal† yet‡).
† Emoji have helped, though HN specifically removes them.
‡ Note, however, that in the HN text input field, the dagger and double dagger are not shown superscript, as they are in the resulting comment you're reading now. It seems to be up to the font to decide what to do, as Unicode opted out of superscripting concerns.
4. paragraph symbol (¶),
5. section mark (§),
6. parallel rules (||),
7. number sign (#).
If more are required, they can be doubled up: double asterisks (**), double single daggers (††), double double daggers (‡‡), etc.
^ one
^^ two
^^^ three
^^^^ four
I think third-party assessment reports like these are a real problem in our industry. There are firms that are worse about them and firms that are somewhat better but it's a near-universal problem.
I don't at all object to technical vulnerability reports being shared. It's good to know when third-party auditors have spotted problems in products (and it's also good for prospective customers of consulting firms to know what kinds of vulnerabilities that firm is likely to spot, and how padded out some of these reports can be with low-quality findings). I also think it's worth remembering that the overwhelming majority of security assessment engagements are never reported out to the public; even when vendors do publish reports, more often than not it's after previous unpublished engagements have been run.
But the way these reports get written creates a huge conflict of interest. They're not simply reports of (1) what was tested and (2) what was found. Too often, they're also product marketing documents, beginning and concluding with "overall assessments" of the target software that is almost invariably positive, even on projects that reduced the target to a smoking crater (to say the least, that didn't happen here, but when it does, it's always framed as "[vendor] has made great strides in improving their security in the wake of this important engagement").
There are firms that specialize in writing public audit reports, and firms that specialize in finding excellent vulnerabilities, and the Venn of those firms is practically two disjoint circles† --- at least in the sense that there's a sort of insider chatter about who the quietly bad-ass firms are, and who the "most credible for shutting down sales objections" firms are.
Even when they're not intended to be tools of persuasion, public audit reports tend to function that way systemically. That's because vendors control the terms on which these audits are done, what's to be tested, when the testing will occur, how much time will be allotted and who's staffing. Vendors pay for public-facing reports, as an extra line item in the SOW, and in some circumstances get to review drafts.
Meanwhile, the public that reads these reports is in no way qualified to weigh or contextualize the report. If you're reading an audit report from a commercial vendor, it is invariably presented as a "clean bill of health". But even if you somehow get principals at Azimuth to assess your system, there is no such thing as a clean-bill-of-health audit. Different teams of auditors will find different bugs (even different teams from the same vendor!).
I think we need a new norm in the industry, and that it can only come from the auditing firms themselves. I think that, roughly, that norm should be that public-facing reports can be provided only in the same dry, technical form they're presented to development teams in ordinary, non-public projects: a methodology, a scope and rules of engagement, and a list of findings. No editorializing and no editorial review by vendors. Probably, though I'm less clear on the mechanics of how this would work, it should also stop being OK to charge different amounts for projects that do and don't have public reports.
I know there are consultants that disagree with me about this; I look forward to reading their takes.
† I'm not editing this out but on reflection this is pretty imprecise and it's probably easy to come up with a counterexample.