This is the imbalanced class problem that comes up often in the real world.

You can't optimize for overall accuracy, otherwise you'd have classifiers that always just predict guilty (or not sick, or not fradulent, and so on) because they'd achieve 95%+ accuracy.

So what is the metric to optimize?

I use AUC often, and always look at the confusion matrix. Metrics based off of it (recall, precision) are both useful to use as well.

In the end the probability cutoffs you choose are more often than not based on business context and costs associated with various actions. For example - if we're trying to predict whether a customer is going to leave or not, I'd choose a different threshold based on whether we're mass e-mailing people, or if we're having customer service reps call people, since the underlying costs for each action are so different.

In the first case it's fine to cast a wide net, and e-mail lots of people who we're less certain about, in the second we'd optimize to target people we're most sure are going to leave, since it costs so much more to reach out to them.

AUC is robust to class imbalance. It can be understood as follows: given an instance of the positive class and an instance of the negative class, it measures the probability of your learned model being able to distinguish between the two.

In my opinion d-prime:

https://en.wikipedia.org/wiki/Sensitivity_index

Maybe mutual information?

Therein lies the problem of the accuracy of predictions. If there's three AI judges, one gives a court case a 35% chance of being guilty, the other a 40%, and the last a 45%. The case was a guilty. Were the predictions correct?

Measuring the accuracy of a single prediction is difficult, but there are reasonable ways to measure the accuracy of a set of predictions. For example, https://en.wikipedia.org/wiki/Brier_score