Huh?
> Furthermore such legacy tests directly hijacking core code come with their own problems, such as breaking all too often (and not because they detected a bug), all while not being able to detect whole categories of bugs.
Pangram is just routinely spammed in every other HN thread, but as far as I can tell, it's not much more reliable than rand()%2, and that's only for cases where the "author" doesn't even care to hide the text was generated.
It doesn't and cannot work in general case. It's trivial to make LLM write in any style, or have it prorduce arbitrary number of typos and grammar mistakes. Most people just don't bother to add an extra line to the prompt.
Several independent audits (not counting Pangram's own research, of course) all finding extremely low or zero false positive rates and pretty good recall:
https://epoch.ai/data-insights/ai-detectors-false-negatives
https://arxiv.org/html/2608.11256
https://link.springer.com/article/10.1007/s40979-026-00226-w
Also, the fact that when a public bounty was placed on a known-human text of >1000 words that Pangram falsely flags as having a >75% AI content, the only example anyone found was this one: https://twitter.com/Nition/status/2122500484007395664
> Pangram is just routinely spammed in every other HN thread, but as far as I can tell, it's not much more reliable than rand()%2
As far as you can tell... based on what?
> It doesn't and cannot work in general case. It's trivial to make LLM write in any style[...]
If you asked me two years ago whether AI detection was possible, I'd have answered no on these grounds. It turns out, however, that RLHF as it's currently done causes LLMs to collapse to very specific styles, and labs are either unwilling or unable to figure out a way to fix it, which is why both humans and models are in fact able to detect AI-written text. Base models don't have this style collapse and indeed aren't detected by Pangram, but very few people are willing to use base models. Similarly, finetuning a model to write in a specific style would probably suffice to evade detection, but very few people are willing to put in this much effort. The result is that it's hard to be sure that any given text isn't AI-written, but most AI-written text is extremely blatant.
Some clues that caught my attention on their own:
“Each test launches a real instance of GIMP and performs a small task. Then it reads the result from GIMP. If the result is incorrect, the test fails and tells us what went wrong.”
That’s classic Claude; just like what I read every day. His approach to humanizing everything
And Pangram IS conservative in its estimates of results. With a text this long, I have no doubt about its accuracy.
> Each test starts a real GIMP and does one small task. Then it reads the result from GIMP. If the result is not correct, the test fails and tells us what went wrong.
Your quote doesn't appear in your Pangram link or in the commit history for the blog https://gitlab.gnome.org/Infrastructure/gimp-web/-/commits/t...
If it went "nope! No AI here!" you'd walk away and wouldn't Pangram your comment with mentioned of Pangram.
Pangram!
I used it because I had strong suspicions about certain passages in this text. Otherwise, I would never have used my Pangram credits on this.
Anyone who regularly reads AI-generated text will quickly recognize the few AI quirks in the text.
Humans can do better. They DO better
Pangram!
It seems hypocritical and raises questions about whether the lead maintainers are following their own guidelines.