we are planning to have a consultant come in a build image comparison of selfies with ID cards (for a fintech mobile app). What kind of validation and variation scores should we ask for to explicitly model the probability ?
I'm also a fan of sensitivity analysis, where you jitter one of the inputs, or one of the model parameters, and see how robust the output is to local noise.
ISV I'm not explicitly familiar with. But there's no reason you couldn't use one of the general purpose techniques from above.
That said, resampling is very computationally intensive, so I can understand why it's not that popular in conjunction with similarly intensive iterative fitting algorithms like backpropagation and MCMC. I didn't mean my post to be a criticism of the research as much as I wanted to just put a warning out there about an issue that researchers without deep statistical backgrounds tend to overlook.
> So, which algorithm to trust?
This is an excellent example of the danger in using algorithms that produce point estimates, but not variability estimates. Whether you model it or not, any inference or prediction exercise implies a probability model (else you wouldn't need to conduct inference at all). Failing to make that model explicit, whether by nested cross-validation or an analytical technique, leads you down a dangerous road of making arbitrary comparisons And breaking near-ties on the basis of "field knowledge", which can turn out to be a little more than glorified gut instinct.