News for hackers. Stuff that matters.
Re:Better Language Models and Their Implications by minimaxir on Thursday February 14, 2019 @01:19PM (#19164328)
The generated text sounds too good; is it possible that the model overfit the source material (especially since the n-previous-tokens value is infinite, while other approaches like char-rnns/textgenrnn use a fixed window length)? It's something I've encountered many times while working with text generation.
Re:Better Language Models and Their Implications by DenisM on Thursday February 14, 2019 @08:31PM (#19167785)
It is uncanny.

I would expect a language model to pull up different person names for each time that one was called for. For a person name to be consistently used through several paragraphs it is not enough to rely on word co-occurrence.

If I had to produce a text like this I would simply take an existing text and replace randomly chosen words with other similar words (as hinted by amvalo). Similar as in - words that tend to occur in similar contexts. So John->Bob throughout entire text. But that would not be a language model product anymore, and where is fun in that?

I should set aside some time to read this paper.

Re:Better Language Models and Their Implications by dannyw on Friday February 15, 2019 @08:52PM (#19176477)
That’s because the transformer model used has memory.
Re:Better Language Models and Their Implications by amvalo on Thursday February 14, 2019 @07:06PM (#19167318)
The unicorn story struck me as being structurally very similar to the standard wire stories about new species discoveries. That's probably why it's the best of the bunch.
Re:Better Language Models and Their Implications by nmca on Thursday February 14, 2019 @01:21PM (#19164343)
Try googling sentences.
Re:Better Language Models and Their Implications by minimaxir on Thursday February 14, 2019 @01:26PM (#19164387)
Hmm, I tested a few sentences and it didn't turn up any exact matches (aside to this article), so maybe I'm wrong.

With a temperature of 0.7/1.0, that's enough for sufficiently random text I suppose. (the raw, uncurated generated text using the smaller model is a bit more random: https://raw.githubusercontent.com/openai/gpt-2/master/gpt2-s...)

Re:Better Language Models and Their Implications by wuthefwasthat on Thursday February 14, 2019 @02:25PM (#19165018)
Those samples are from the large model (GPT-2)! Regarding memorization vs. generalization, see our paper for more analysis.
Re:Better Language Models and Their Implications by yorwba on Thursday February 14, 2019 @04:22PM (#19166168)
Have you tried to determine which parts of the training data contributed to the model generating a certain output? I wonder whether the model avoids reproducing exact matches of the training data by splicing several similar articles together.

For example, the generated text about the Civil War mentions that Thomas Jefferson Randolph [0] was named after his grandfather, the president. But is the wording mostly influenced by articles talking about that specific fact, or does it draw from more general examples of someone being named after their grandfather?

[0] https://en.wikipedia.org/wiki/Thomas_Jefferson_Randolph

Re:Better Language Models and Their Implications by zalo on Thursday February 14, 2019 @05:03PM (#19166502)
Does it seem like there will be any way to go backwards from the sample to the prompt?

From a safety perspective, it would be useful to see what prompt a piece of text might have been generated with...