Yes! That is what is scary as fuck with these "magic classification" techniques. The more reliable they are, the more people dismiss false positives. This, applied to criminality is totally 1984.

For those that don't know, "this applied to criminality" already exists and has some problems.

Propublica wrote a story criticizing it's use: https://www.propublica.org/article/machine-bias-risk-assessm...

The company responded with a rebuttal: https://www.documentcloud.org/documents/2998391-ProPublica-C...

And Propublica has a counter-rebuttal: https://www.propublica.org/article/propublica-responds-to-co...

Propublica has an "interesting" definition of "fairness". It turns out there is a beautiful mathematical result that shows that the different definitions of "fairness" are inherently incompatible with each other and cannot be satisfied at the same time.

This is the best article that summarizes the controversy on this subject:

https://www.washingtonpost.com/news/monkey-cage/wp/2016/10/1...

Paper by Jon Kleinberg and Sendhil M. and Manish Raghavan on this topic https://arxiv.org/abs/1609.05807

What happens where these kinds of models are the reason for suicidal thoughts or criminality?

Quite right. I would have expected any paper to include both.

This is quite a good article on the subject: https://en.wikipedia.org/wiki/Sensitivity_and_specificity

The linked article contains exactly this information, though. Figure 1 showed the Receiver Operating Characteristic curve, which they called something else, but it's the standard way of discussing this. It plots the classifier's true positive rate versus the false positive rate at various settings thusly: the y-axis plots the "sensitivity", which is the true positive rate. The x-axis plots the "specificity", which is equal to (1 - false positive rate)... but because everyone is thinking of the false positive rate anyway, you'll notice that the specificity goes from 1 to 0.

Incidentally, these are plotted as a curve because it's trivial to build a classifier with a 100% true positive rate (call them all suicidal) or a 0% false positive rate (call none of them suicidal), but just how "good" your model / algorithm is is a function of how it performs with both.

> P(+|suicidal) = .93 > I want to know P(suicidal|+)

Can someone explain what the difference between these 2 is? I'm not versed on statistics.

I believe you read the | as "Therefore" (possibly going for ⊢ ?), as in "The probability that someone who scored highly on this test is also suicidal is .93" They want to know that the probability that someone who is suicidal scores highly on the test. The test could be good at identifying suicidal thoughts in only a small portion of the population, since they have demonstrated a good false-positive rate (i.e. 93% of subjects that it identifies as suicidal are); but not good at identifying which members of a population are suicidal - that is, if given to all of a population, only find some small percentage of people with suicidal thoughts.

Read the pipe as "given" (or "when"). P(A|B) = "probability of A given B".

You've got people who are actually suicidal. You've got people for whom the test comes back positive. There are two subtly yet crucially different metrics. The paper reports the answer to: if someone is positive, will the test come back positive? The flipside is very important for customers of the test: if the test comes back positive, does that mean I am suicidal?

P(+|suicidal) = the probability that the test comes back positive given the patient is actually suicidal = 0.93

P(suicidal|+) = the probability that the patient is suicidal, given that the test came back positive.

As a degenerate case, imagine Nat's Suicidal Tendency Detector.

10 PRINT "SUICIDAL"

It will correctly detect every single suicidal patient put in front of it. P(+|suicidal) = 1. IT'S A MIRACLE BREAKTHROUGH!

That test's critical failure is P(suicidal|+) ... it's identical to the rate in the population. As you could have guessed by reading the source code, taking the test doesn't give you any more information about the patient's suicidal tendencies.

Whew. Hope that was coherent! Google "conditional probability" if you want the math of how to work with these.

Thank you! I think I understand now, this clarifies a lot!