Now the novelty is that this can be better targeted. But even simple Markov-chain based text generators were good enough to fool people for a bit.

And there was always people that had too much free time to write. A lot. (See for example the crackpots and conspiracy theorists that bombard physics forums. See the 9/11, Zeitgeists, etc. movies. See how much has been written about anti-vaxx, about quantum woo, etc.)

Reputation systems work pretty well for countering spammers.

And against APTs (advanced persistent threats, spearfishing attacks, etc) there's no real "universal" protection anyways. (You need a competent security team to out think and out resource the attackers in every possible dimension.)

This AI is the same as the paid Russian trolls and the unpaid scammers, and so on.

I agree with your last point though - it falls into the same category as paid Russian trolls. I think that's exactly why they were hesitant to release the pre-trained models - they didn't want to make it easier/cheaper for a bad actor to replicate the 2016 election.

It remains to be seen whether their decision will make an iota of a difference. But I understand their motivation.

I work in this field, and yes, this is very novel (at least in terms of the quality).

It's the biggest improvement in quality I've ever seen. The long term coherence is so much better than anything else that has ever been built.

Yes, it can serve as great customized propaganda generator, and yes, people can be spin 'round and 'round with it. But they can be already with pretty much anything, from the simplest of phrases from "make X great again" to the elaborate scams of new age bullshit.

I simply disagree on the "virulence" or weaponization factor of this with others. (Especially when it comes to the possible "defenses", none can be "deployed" in 6 months. You can't teach critical thinking to billions of people overnight.)

I don't have a strong opinion about if they should have released this model or not.

I do know it would make a great commercial spam generator though. Want a million product reviews which seem legitimate quickly? This is the thing..

Conjecture: GPT-2 trained on reddit comments could pass a "comment turing test", where the average person couldn't distinguish whether a comment is bot or human with better than, say, 60% accuracy.

At this stage, these AI's can only help. Imagine we are given this tool that can generate samples from the "uninformative but realistic looking text" distribution, we can then put it in a discriminator to filter out blabbering bots and humans together, or invert it to summarize the small kernel of information, and that would be a great thing. The better these models learn about typical human behavior the better off we are at identifying the truly exceptional. It's when AI starts to sense and incorporate novel information from the non-human environment that you really have to worry.

Perhaps, but that's the world we live in. I suspect the average reddit commenter is already more articulate than the average person (citation needed, I know. But reddit skews highly educated young male in a first-world country. There's no way they do worse than a worldwide average).

Other than that, I agree with your comment.

But it's no symbolic reasoning. It's not constructing a counter-argument from your argument. It simply lives off previous epic rap battles of internet flamewar history about .. well, about anything, since it's the Internet, and people like to chat, talk, write essays on every topic there is. Satire too. So there is always something to build that lang model on.

Though that will come too. Eventually.

There is no real profit to be made by generating realistic looking text. Spammers don't work that way, spammers haven't cared about realistic looking text for years. Nor have spam filters cared much about text for a long time, exactly because it's so easy to randomise. Anti-spam is not a good reason to hold back on language generation models, in my view.

As for HN, if bots can write posts as good as humans, great, why hold back?

People are not trivial automatons who can have their opinions rewritten on the fly by auto-generated text. If auto-generated text reaches into its giant grab-bag of learned expressions and produces something actually interesting or insightful, people might be interested in that new line of thinking, but if - like many of these examples - it's essentially rambling if coherent nonsense then it won't have any impact at all.

So I rather think it's you fooling yourself. You've been reading comments online for years without knowing who or what produced them. If you discovered half of them were artificial tomorrow, what difference would it make? The people around you are already judging arguments based on the content, not their volume or who wrote them.

It seem that as far as information warfare goes “less is more” works quite well and they rely on targeted people to spread the news for them.

When you want to drive an agenda you don’t need unique 100,000 comments you need a good copy pasta.

Overall I’m sick of this dramatization of the AI catastrophe until there will be a proven path with agency for it to actually operate in the real world.

A chat bot isn’t a threat to anyone even if it turns homicidle.

Regarding an agenda, sure, good pasta is fine and all, and regular ol people are fine, but it is not cost effective. This is a million times cheaper, which means you can use it everywhere, not just the obvious places, you can be everywhere, and you can do more than just push a couple big items, you could push tens of thousands of them, micro targeted all the way down to the individual. Don’t dismiss it so easy—the potential scale is far, far larger than anything existing to date.

And I would note that the reason 100,000 comments aren’t effective now is precisely because they are too formulaic, too obviously fake when used on such a large scale. This has the potential to create real, live, seemingly active and believable online communities of millions of people, all at fractions and fractions and fractions of a penny compared to current methods. People read news, then comments (or reviews or whatever), because they use them to determine the validity of the content they just read; if it’s no longer possible to tell from the comments what’s a scam and what isn’t... well, you could do a lot of things with that.