Re:Ask HN: Should HN ban ChatGPT/generated responses?
by carbocation on Sunday December 11, 2022 @11:33PM
(#33950906)
I would argue that this test isn't particularly informative. Given 5 attempts and 5 successes, even though the point estimate of accuracy is 1, the 95% CI ranges from 0.48 to 1:
In other words, we don't have enough data in that small sample to reject the possibility that the model is 50% accurate, much less 99.9% accurate.
[ Parent
| on HN ]
Re:Ask HN: Should HN ban ChatGPT/generated responses?
by viraptor on Sunday December 11, 2022 @11:54PM
(#33951022)
I think the message was claiming something else, specifically that each classification was given a score of how confident the model was in the answer and the answers were given 99.9%+ in those cases.
See the app: https://huggingface.co/openai-detector/ - it gives a response as % chance it's genetic or chat bot.
Re:Ask HN: Should HN ban ChatGPT/generated responses?
by ivegotnoaccount on Monday December 12, 2022 @04:39AM
(#33952766)
Seems to have major biases against who knows what sentence structures. Even without trying to make it say fake, some of my messages and text I write in it are pretty confident I'm GPT-2...
Re:Ask HN: Should HN ban ChatGPT/generated responses?
by carbocation on Monday December 12, 2022 @09:27AM
(#33954929)
With 5 samples, we have no way to assess whether the app’s 99.9% self-assessment is remotely well calibrated. (As noted above, 5/5 is also consistent with a model that is right 50% of the time.)