Re:Writing for Snobs
by gwern on Tuesday September 15, 2020 @09:49PM
(#24488777)
"drowned out in the noise" is not really a thing here.
Style is just another aspect of text it learns to model and predict. If there are style cues, it makes that part of the prediction and continues it; if there are no style cues, then it just starts somewhere, perhaps with some style common on average...
But if there's a decent amount of text clearly linked to a specific author or style, it will learn the quirks, the vocab, the topics, and so on. Shakespare, no problem, of course it can do Shakespeare pastiche - it can do Paul Graham pastiche, it can even do gwern pastiche! Read through the samples in https://www.gwern.net/GPT-3 and tell me again that everything is just "drowned out in the noise" and GPT-3 is unable to produce anything but a bland average style...
Style is just another aspect of text it learns to model and predict. If there are style cues, it makes that part of the prediction and continues it; if there are no style cues, then it just starts somewhere, perhaps with some style common on average...
But if there's a decent amount of text clearly linked to a specific author or style, it will learn the quirks, the vocab, the topics, and so on. Shakespare, no problem, of course it can do Shakespeare pastiche - it can do Paul Graham pastiche, it can even do gwern pastiche! Read through the samples in https://www.gwern.net/GPT-3 and tell me again that everything is just "drowned out in the noise" and GPT-3 is unable to produce anything but a bland average style...