News for hackers. Stuff that matters.
Re:Can a Machine Learn to Write for the New Yorker? by grenoire on Monday October 07, 2019 @09:40AM (#21180632)
> In the event, I’d been able to persuade GPT-2 to express the thoughts that I wanted it to express, although I didn’t feel any more confident in my own knowledge of language or my own prowess at predicting what it might say. After a few minutes, I went on to the next page of GPT-2’s work, and the computer revealed a series of explanations about why words used in “Atlas Shrugged” had apparently been chosen. “Liking is true universal,” the machine said. “It applies to all individuals.” “Everybody is drawn to success,” it said. “Not only do they do what they want, but, to the extent that they are successful, they also help others.” The most interesting part of the text, however, was the last line: “You don’t have to like it. But you should care.”

I like how in this generated piece, the model writes as the author, and quoting the model itself has some form of a reflective second-person view of itself. Quite hilarious and hella' sci-fi.

Re:Can a Machine Learn to Write for the New Yorker? by sparsely on Monday October 07, 2019 @09:42AM (#21180657)
The generated text is a little too on the nose and sure enough:

> In each case, we generated more than one response and selected the predicted text that follows each section in the article.

Good article even so.

Re:Can a Machine Learn to Write for the New Yorker? by hammock on Tuesday October 08, 2019 @06:06PM (#21197801)
The cheat you point out IS the state of the art for predictive text, though. When I'm texting on my Google phone, or emailing from Gmail, it gives me 3 possible options to choose from.
Re:Can a Machine Learn to Write for the New Yorker? by hyperbovine on Tuesday October 08, 2019 @11:43AM (#21193496)
Agree, I was blown away by the quality of the generated text until realizing that it was simply too good to be true. The fact that they picked and chose makes for a way less impressive demo. Good piece though.
Re:Can a Machine Learn to Write for the New Yorker? by gwern on Tuesday October 08, 2019 @01:59PM (#21195073)
But on the other hand, 'picking and choosing' can be automated in many ways (eg A/B testing), and in a very real sense, that's what the last research OA released on GPT-2 is about: https://openai.com/blog/fine-tuning-gpt-2/ (Because the human preferences are used to train a critic which then 'picks and chooses' by ranking output by quality to further train the text generator.)
Re:Can a Machine Learn to Write for the New Yorker? by gwern on Sunday October 13, 2019 @10:35AM (#21240567)
I have been experimenting with the preference-learning code for writing poetry, and it does seem to work: I've already noticed improvements in the generated poetry in increasing the amounts of rhymes, avoiding undesirable text genres like Greek/Latin/Spanish/English-prose, and decreasing the amount of repetition. (One of the most irritating and longstanding failure modes in neural text generation is that it often degenerates into repeating the same phrase many times, so it's pleasing to see that penalizing this in ratings does seem to then cause the generator to learn to avoid it.) Unfortunately, model-free RL training using PPO is extremely slow. Fortunately, I can't see any reason that it has to be PPO or model-free RL at all, since the reward model ought to be fully differentiable and allow calculation of true gradients / model-based RL, so OOM speedups ought to be possible quite easily.
Re:Can a Machine Learn to Write for the New Yorker? by wmhorne on Monday October 07, 2019 @10:57AM (#21181612)
> Unlike writing, speech doesn’t require multiple drafts before it “works.” Uncertainty, anxiety, dread, and mental fatigue all attend writing; talking, on the other hand, is easy, often pleasant, and feels mostly unconscious.

While I like the article, this distinction feels overly anecdotal and subjective to me.

Re:Can a Machine Learn to Write for the New Yorker? by chaorace on Tuesday October 08, 2019 @07:31AM (#21191068)
I agree. Most people would sound quite stupid if their speech were to be transcribed as-is. I feel that It's not that we are more skilled speakers than writers, but more so that we hold writing to a higher standard in general.
Re:Can a Machine Learn to Write for the New Yorker? by qplex on Tuesday October 08, 2019 @12:10PM (#21193811)
Adding to this: it's not just the textual content. Speech almost always carries an additional layer of information, imprinted by who or whatever spoke the words.
Re:Can a Machine Learn to Write for the New Yorker? by mamon on Monday October 07, 2019 @11:12AM (#21181787)
Judging by the quality of writing on web portals in my country this has already happened (although it is possible that most of those articles are just machine translations of pieces from UK/US websites)
Re:Can a Machine Learn to Write for the New Yorker? by RickJWagner on Monday October 07, 2019 @09:15PM (#21187731)
"Along with almost everyone else who texts or tweets, with the possible exception of the President of the United States, I have long relied on spell-checkers and auto-correctors"

This exactly spells out a question I have. How would the New Yorker make sure it's AI author puts the correct amount of Left-snark on what it writes? How would Fox news tune theirs to Right-snark?

Re:Can a Machine Learn to Write for the New Yorker? by brummm on Tuesday October 08, 2019 @11:48AM (#21193553)
Seems more like objective-snark instead of left-snark to me...