Re:Bot Submissions to Comment Website Can’t Be Distinguished...
by notahacker on Friday December 20, 2019 @09:05AM
(#21844162)
I'm also wondering whether the 'handpick[ed]...to ensure a high coherence and high relevance' GPT-2 comments actually outperform the comparatively trivial sentence-spinning script in getting approved by MTurkers.
Think https://www.reddit.com/r/SubSimulatorGPT2/ is more impressive than a study where half of GPT-2 comments handpicked for being human-like by one human were accepted by another human. Particularly given that some of the comments in question were three or four words long...
> To quickly weed out inappropriate comments, I handpick from generated comments those that ensure a high coherence and high relevance sample for submission.
So basically it's a validation of GPT-2 making sense with small amounts of text. Judging from the demo test page, they are pretty good texts, but he said it himself that larger texts betray the bot. So, i m not sure what he's trying to prove by using MTurkers, since this does not attack the problem mentioned in his introduction: the fake FCC comments were weeded out through text analysis, not via human work.
In all, i'm not sure if this is something that people didn't know about gpt-2. The title is certainly not justified, perhaps "Curated bot comments can't be distinguished by humans to be obviously fake" would be better, but also more banal.