I remember an article which showed that GPT-3 quality of answers were very dependent of the previous questions which I find quite surprising as it has already been trained on a large number of texts..

And there were an attempt to train GPT-2 with a common sense database: https://www.quantamagazine.org/common-sense-comes-to-compute... it would be interesting see the results with GPT-3..