Re:Ask HN: Should HN ban ChatGPT/generated responses?
by carbocation on Sunday December 11, 2022 @11:33PM
(#33950906)
I would argue that this test isn't particularly informative. Given 5 attempts and 5 successes, even though the point estimate of accuracy is 1, the 95% CI ranges from 0.48 to 1:
> binom.test(5,5,0.5)
Exact binomial test
data: 5 and 5
number of successes = 5, number of trials = 5, p-value = 0.0625
alternative hypothesis: true probability of success is not equal to 0.5
95 percent confidence interval:
0.4781762 1.0000000
In other words, we don't have enough data in that small sample to reject the possibility that the model is 50% accurate, much less 99.9% accurate.
Re:Ask HN: Should HN ban ChatGPT/generated responses?
by viraptor on Sunday December 11, 2022 @11:54PM
(#33951022)
I think the message was claiming something else, specifically that each classification was given a score of how confident the model was in the answer and the answers were given 99.9%+ in those cases.
See the app: https://huggingface.co/openai-detector/ - it gives a response as % chance it's genetic or chat bot.
Re:Ask HN: Should HN ban ChatGPT/generated responses?
by ivegotnoaccount on Monday December 12, 2022 @04:39AM
(#33952766)
Seems to have major biases against who knows what sentence structures. Even without trying to make it say fake, some of my messages and text I write in it are pretty confident I'm GPT-2...
Re:Ask HN: Should HN ban ChatGPT/generated responses?
by carbocation on Monday December 12, 2022 @09:27AM
(#33954929)
With 5 samples, we have no way to assess whether the app’s 99.9% self-assessment is remotely well calibrated. (As noted above, 5/5 is also consistent with a model that is right 50% of the time.)
Re:Ask HN: Should HN ban ChatGPT/generated responses?
by mattnewton on Sunday December 11, 2022 @01:35PM
(#33945951)
Not that the HN team isn’t capable of solving this, but I think that would be a pretty significant increase in server cost to run every comment through a classifier.
Re:Ask HN: Should HN ban ChatGPT/generated responses?
by system2 on Monday December 12, 2022 @03:53AM
(#33952443)
We don't have millions of comments here. I think a $5 additional vcpu can solve it easily.
Re:Ask HN: Should HN ban ChatGPT/generated responses?
by mattnewton on Monday December 12, 2022 @04:55PM
(#33960981)
My gut tells me there is no way a single cpu model can quickly spit out useful scores here
Re:Ask HN: Should HN ban ChatGPT/generated responses?
by rippercushions on Monday December 12, 2022 @12:02AM
(#33951074)
The GPT detector requires a significant amount of text for accuracy. Works well on student essays, not so much on short comments.
Re:Ask HN: Should HN ban ChatGPT/generated responses?
by ivegotnoaccount on Monday December 12, 2022 @04:35AM
(#33952744)
https://news.ycombinator.com/item?id=32447928 is marked as nearly 100% fake, whereas I can assure you it was written by a human.
Maybe I was just unlucky with the comment I tried it with (took the longest one I saw in my history), but I don't think I would have liked seeing it either removed or spat at for being considered as "AI generated"...
The detector also thinks this comment is fake. Seems influenced by flavors of mistakes.
Idiomatic ones. Spelling ones. Grammar. All non-native speakers will easily get flagged. Does not look spot-on for now. Checked all those assertion live-typing on the demo. 0.09% real.
Re:Ask HN: Should HN ban ChatGPT/generated responses?
by seydor on Sunday December 11, 2022 @01:38PM
(#33945983)
It would work, for like 1 day
Re:Ask HN: Should HN ban ChatGPT/generated responses?
by arbitrage on Sunday December 11, 2022 @01:45PM
(#33946058)
HN doesn't update anything. That will obviously never happen.
Re:Ask HN: Should HN ban ChatGPT/generated responses?
by dang on Sunday December 11, 2022 @11:32PM
(#33950897)
We update things all the time! But we prefer to be subtle about it.
> The @huggingface GPT detector works very well on ChatGPT-created text. I ran 5 student essays and 5 ChatGPT essays for the same prompt through it, and it was correct every time with >99.9% confidence.
How about adding a %human/%GPT statistic to posts and comments?