News for hackers. Stuff that matters.
Re:Turing-NLG: A 17B-parameter language model by gchq-7703 on Tuesday February 11, 2020 @04:28AM (#22297174)
I'm fairly sure that these models don't work solely on the previous word, but instead are able to remember some level of information from history.

Otherwise, you'd reach a word like 'and' and couldn't possibly follow it with a logical statement that follows on from the previous part.

Re:Turing-NLG: A 17B-parameter language model by ragebol on Tuesday February 11, 2020 @05:51AM (#22297517)
This is why I said 'words', multiple :-).

My point being that these generation models should be conditioned on something more than just word history, like something they want/are instructed to express.