New AI classifier for indicating AI-written text

Re:New AI classifier for indicating AI-written text by claytonjy on Tuesday January 31, 2023 @02:08PM (#34600154)

Is there a longer-form paper on this yet? TPR (P(T|AI)) and FPR (P(T|H)) are useful, but what I really want is the probability that a piece flagged as AI-generated is indeed AI-generated, i.e. P(AI|T). Per Bayes rule I'm missing P(AI), the portion of the challenger set that was produced by AI.

If we assume the challenger set is evenly split 50-50, that means

    P(AI|T) = P(T|AI)P(AI)/P(T) = (0.26)(0.5)/(0.26+0.09) ~ 37%

So slightly better than a 1/3 chance of the flagged text actually being AI-generated.

They say the web-app uses a confidence threshold to keep the FPR low, so maybe these numbers get a bit better, but very far from being used as a detector anywhere it matters.

[ Parent | on HN ]

Re:New AI classifier for indicating AI-written text by TchoBeer on Tuesday January 31, 2023 @04:38PM (#34602590)

>Per Bayes rule I'm missing P(AI), the portion of the challenger set that was produced by AI

This will obviously depend on your circumstances.

[ Parent | on HN ]

Re:New AI classifier for indicating AI-written text by drc500free on Tuesday January 31, 2023 @10:20PM (#34605909)

Precision is impossible to calculate without knowing P(AI), which is use-case specific.

Source: Spent 10 years trying to explain this to government people who insisted that someone tell them Precision based purely on the classifier accuracy without considering usage.

[ Parent | on HN ]