It’s smoke and mirrors trying to fool you into thinking it’s generating intelligent text. In some applications e.g., a chatbot, that’s appropriate. But it’s really no comparison to an expert system for most applications, where you know exactly the right and wrong solutions. Not adding numbers correctly with the huge budget GPT3 has for training and inference is a poignant case of that fact. A linear layer taking in x and y will learn x+y just by setting the weights to 1.0, so it’s not even a hard problem for neural nets, just in the particular tokenization and architecture used for GPT models.