I think earlier versions (GPT2? 3?) were a bit wonky in the sense that they'd make sentences which appeared grammatically correct but not actually make sense. But I didn't play around with them too much. Anyway, the article was nearly the same in 2020 so it's unlikely to be that.