From the other side of the fence: I've been hiring companies to do external pen tests for fifteen years now. Some of them have been giant corporations with security divisions, some of them have been just past the startup stage, and some of them are recognizable big names in the industry. I've signed one year, two year and three year contracts.

A few of them have distinguished themselves, slightly, in the first year of a multi-year contract -- and then regressed to the mean in the rest.

Quoted prices vary by a factor of 4, approximately. Work done does not. Quality of work appears to be basically independent of price.

Arrogance scales with price, though.

Even if you are a non-checklist firm, I can't justify hiring you at a higher price because I can't differentiate you from the bloviators, and basically nothing you say other than "I am ptacek" can change that.

Security unfortunately is a creative process, which is hard to get consistent. I would love to use something like statistical product control, but I have yet to see a good way to apply it (or similar techniques) without forcing pen-tests to be “follow the checklist”.

I should add that burnout is a massive problem when it comes to quality. Even if you found a good consultant (or group of), for round #2, they could have just finished testing all of your competitors products and are now worn out.

Scheduling in variety to try and prevent partial burnout is difficult if everyone in a vertical only wants to use the same person.

This is a good perspective. I pushed for a higher priced pen test firm in our last engagement due their name/reputation (i was sick of nessus scan outputs) but ultimately their price was just SO MUCH cheaper than the prestigious named one. I couldn't provide a justification to leadership outside of "they have these awesome writeups in their research division". Asking to see a sample report to prospective firm turns out to be a crapshoot as well (but seems to be one of the few things to go off of). Can you provide any insight on how you evaluate?

The pentesting industry has a big problem with being a good "market for lemons".

It's very hard for customers to differentiate the good and bad companies, without having their own internal expertise, and even then you need to go down the line of getting named testers and speaking to each one.

Another problem is with how many/most pentest companies report, which is by exception. There's no requirement to state all the tests they did, just the results, so it's hard to tell the difference between "we've got a good system and they didn't find much" and "they didn't do good work and missed things"

Our reports always have a section on things we looked at or for. Doing so takes a majority of the time it takes to write the report. Since it adds to the cost, firms that don’t do this can underbid the ones that do.

For slimmed down reports that are published/given to third parties (given to the customer’s customer) it’s extremely important that the scope of the test is detailed in that letter. The charlatan firms will be happy to omit the fact that they only tested the “about us” page while blind-folded.

So if you are reading a “letter of assessment” for something you are thinking of adopting, just look for the scope of the test, and if it sounds reasonable they potentially had a good test. If it’s missing, the test wasn’t worth the electrons in that letter.

This is what I am working hard towards, moving to a firm like yours.