News for hackers. Stuff that matters.
Re:OpenAI Trains Language Model, Mass Hysteria Ensues by SubiculumCode on Sunday February 17, 2019 @02:47PM (#19186405)
To what extent is this not just finding text samples written in its training sample and regurgitating it near verbatim?? -Non ml guy
Re:OpenAI Trains Language Model, Mass Hysteria Ensues by HALtheWise on Sunday February 17, 2019 @03:23PM (#19186644)
If you look at their paper, section 4 is entirely devoted to this question. They present compelling evidence that it is generating original content, the simplest of which is it's ability to write coherently about ridiculous things like talking unicorns that nobody has ever written about in the training set.

https://d4mucfpksywv.cloudfront.net/better-language-models/l...

Re:OpenAI Trains Language Model, Mass Hysteria Ensues by arcticfox on Sunday February 17, 2019 @03:39PM (#19186735)
The talking unicorns piece was shockingly good. That is at least as coherent of a news story than the average human could easily invent about it.

Reading that piece gives me the same weird feeling as watching AlphaStar playing through a StarCraft game.

Re:OpenAI Trains Language Model, Mass Hysteria Ensues by applecrazy on Sunday February 17, 2019 @02:50PM (#19186427)
You bring up a good point. Without seeing their code and training metrics, how do we know that this isn’t some extremely overfitted model?
Re:OpenAI Trains Language Model, Mass Hysteria Ensues by vedant on Sunday February 17, 2019 @08:05PM (#19187973)
From the paper:

"All models still underfit WebText and held-out perplexity has as of yet improved given more training time."