In fact I believe this tool is even more worrisome, because there are a very large number of non-tech savvy people who express their dissonant opinions simply under the mask imparted by internet anonymity. I imagine most everyone here on HN have at least once made an anonymous account to post a comment somewhere that they would rather not have tied to their identity. This is an avenue that is necessary for the preservation of free speech. Remember that things like treating black people as equals, giving women the right to vote, gay rights, etc were and in some ways still are taboo subjects that bring the wrath and ire of the power du jure.
All that said, I still don't see any reason why such a tool shouldn't be released. Why? Even though I believe the tool to be harmful, it's better to know that it exists, know its capabilities, and most importantly know how it works. It'll end up in the hands of the wrong people anyway, so it's better to at least get it into the hands of the right people who can possible do something to combat it.
This is a really important point. Even if you don't like the possible uses of the tool, it's either "release it now and make it possible to defend against" or "don't release it, and hope the likes of the NSA don't develop their own version".
Orwellian.
Ask anyone who's ever lost an ebay or a google account to an algorithmic burp and was essentially banned for life without appeal or even human oversight.
As in, flames everyone to a crisp, posts as much porn as possible, tries to incite a civil war between a few members that might not be on good terms with each other or the staff and registers hundreds of accounts, some of which stay semi dormant until they strike?
Because that can happen very easily online, especially if you get the ire of someone with a lot of free time and very few morals. Or if your site ends up at war with a troll site/gets raided by 4chan.
Do you avoid the hassle now, or wait until the situation blows up and half the site is now in the middle of it?
I voted to release the tool, but you've set up this false scenario with the intention of knocking it down easily and discrediting the opposing argument.
The idea that this tool would be useful for community management is terrible.
I'm a moderator of multiple online spaces.
A few months ago, in one of them, a user got too heated and started flinging insults at someone else. As was standard policy for the place where it was occurring, I issued the user a ban of a few days (enforced cooling-off) and pointed to our guidelines on how to behave.
This user then proceeded, over a period of months, to continually harass me, send me increasingly graphic threats, and try to track me down in real life.
Pray tell, how exactly would you go about "converting" such a person to be productive? I come to you since you are apparently quite the expert on it, or else you wouldn't be giving out advice to just "convert" people.
I don't know how well that would work for high-traffic forums of today, but it can scale pretty easily by employing many moderators.
Unfortunately if the tech exists, it will be released, but I don't think the positives outweigh the negatives here. The chilling effects in terms of comments alone would be bad.
1 and 2 could both be a bet negative for humanity just as easily as they could be a positive.
Or maybe an odd case where it turns out the author of a book or creator of a product finds out someone they know in real life left the negative review and physically attacks them or something.
I mean sure, there are various 'immoral' uses for it (like doxxing), but there are also many good ones. Such as:
1. Working out who wrote a bunch of anonymous reviews on Amazon or other such sites, which could be used to stop fake reviews. You actually mention this usage in your article.
2. Being able to identify troublemakers in a community (such as a forum or a social networking site). I'm sure a lot of administrators would love to know if that suspicious looking new guy is the alias of a banned troll from a few weeks back (posting through a proxy server).
3. Literary analysis, like working out who wrote many anonymous works of fiction. Or as someone said below, determining which parts of Shakespeare plays were actually written by Shakespeare.
4. Crime solving. If it works anywhere near as well as you say, it could theoretically help unmask the Zodiac Killer, or perhaps even Jack the Ripper (if any of those letters were real).
All the uses above would be a net positive for humanity, and would be great possible uses for a deep learning tool like this.
Don't let the worries about its usage by 'bad' people overshadow the good you can do by releasing it.