Lin, B. Y., Chen, X., Chen, J., & Ren, X. (2019). KagNet: Knowledge-Aware Graph Networks for Commonsense Reasoning. 2822–2832. https://doi.org/10.18653/v1/d19-1282
Liu, W., Zhou, P., Zhao, Z., Wang, Z., Ju, Q., Deng, H., & Wang, P. (2019). K-BERT: Enabling Language Representation with Knowledge Graph. Retrieved from http://arxiv.org/abs/1909.07606
Trinh, T. H., & Le, Q. V. (2019). Do Language Models Have Common Sense? Iclr, 1–12.
Ostendorff, M., Bourgonje, P., Berger, M., Moreno-Schneider, J., Rehm, G., & Gipp, B. (2019). Enriching BERT with Knowledge Graph Embeddings for Document Classification. Retrieved from http://arxiv.org/abs/1909.08402
What do you mean by this? Of course they do, learned from their training data. For example, here is quote from conversation 38 of https://github.com/google-research/google-research/blob/mast...
Human: Do you like Korean food in general? Meena: It's okay. I like beef bulgogi, but I'm not a huge fan of kimchi.
It seems to me Meena "knows" bulgogi and kimchi are Korean foods. Isn't that common sense? If it isn't, what do you mean by "common sense"?
Human: Do you like Korean food in general?
Meena: It's okay. I like beef bulgogi, but I'm not a huge fan of kimchi.
Human: Ok what should I shop for ?
Meena : You've got almost everything but you need a pear, the steak and some ginger.
The problem with language models as commonsense is that they are collections of patterns and associations, and that they don't have inference models or solvers - unlike my dog for example!
The fundamental limitation of these "optimization tools" as you call them - they don't have any common sense, and any way to query an external source of information (e.g. wikipedia), or ask a human to clarify.
Another big problem is we don't have any way to do quality filtering on the outputs. From my experiments with GPT-2, it produces one interesting paragraph of text out of 20 - if you squint at it really hard. And most of those 20 don't make much sense at all.
So no, the existing ideas are definitely not enough. Maybe some novel hybrid of symbolic AI with statistical optimization will lead to a breakthrough. This one does not strike me as anything other than "let's use moar weights!!"