Let's take a question answering example. Take just about any recent deep learning paper and try to answer detailed, higher level questions against it. To use a concrete example, take MobileNet V3 paper and ask your system "do I put activation before or after squeeze and excitation" (correct answer is "before"), or "do I need a bias in squeeze and excitation module" (correct answer is "it depends on the task"). You won't be able to, because a lot of things are just _assumed_, just like in any other realistic example of text written for human consumption. The facts are encoded externally as information about the world, and they're so fine grained and contextual, that we don't even know how to begin incorporating them into the answers, let alone do so contextually and probabilistically, like human mind does.

I’m optimistic because I believe that the contextual information that you are describing, is already there in the vast expanse of the internet.

But I will also add, I think none of this will spawn AI, just that it will spawn new technologies that are categorically different.

The fundamental limitation of these "optimization tools" as you call them - they don't have any common sense, and any way to query an external source of information (e.g. wikipedia), or ask a human to clarify.

Another big problem is we don't have any way to do quality filtering on the outputs. From my experiments with GPT-2, it produces one interesting paragraph of text out of 20 - if you squint at it really hard. And most of those 20 don't make much sense at all.

So no, the existing ideas are definitely not enough. Maybe some novel hybrid of symbolic AI with statistical optimization will lead to a breakthrough. This one does not strike me as anything other than "let's use moar weights!!"

Lin, B. Y., Chen, X., Chen, J., & Ren, X. (2019). KagNet: Knowledge-Aware Graph Networks for Commonsense Reasoning. 2822–2832. https://doi.org/10.18653/v1/d19-1282

Liu, W., Zhou, P., Zhao, Z., Wang, Z., Ju, Q., Deng, H., & Wang, P. (2019). K-BERT: Enabling Language Representation with Knowledge Graph. Retrieved from http://arxiv.org/abs/1909.07606

Trinh, T. H., & Le, Q. V. (2019). Do Language Models Have Common Sense? Iclr, 1–12.

Ostendorff, M., Bourgonje, P., Berger, M., Moreno-Schneider, J., Rehm, G., & Gipp, B. (2019). Enriching BERT with Knowledge Graph Embeddings for Document Classification. Retrieved from http://arxiv.org/abs/1909.08402

What do you mean by this? Of course they do, learned from their training data. For example, here is quote from conversation 38 of https://github.com/google-research/google-research/blob/mast...

Human: Do you like Korean food in general? Meena: It's okay. I like beef bulgogi, but I'm not a huge fan of kimchi.

It seems to me Meena "knows" bulgogi and kimchi are Korean foods. Isn't that common sense? If it isn't, what do you mean by "common sense"?

Human: Do you like Korean food in general?

Meena: It's okay. I like beef bulgogi, but I'm not a huge fan of kimchi.

Human: Ok what should I shop for ?

Meena : You've got almost everything but you need a pear, the steak and some ginger.

The problem with language models as commonsense is that they are collections of patterns and associations, and that they don't have inference models or solvers - unlike my dog for example!

A more relevant analogy might be a talking parrot :)

I don't think anyone familiar with the area thought that ConvNets will give us AGI.

However, their effect has been huge! It's hard to overstate this. Computer vision used to be a small niche topic, with tons of effort required to get something working even on simple images. The quality of today's ConvNet predictions is way beyond anybody's imagination in around 2010. Models built around that time were like a house of cards. Extremely carefully crafted for specific scenarios, where moving one threshold a bit would destroy your output.

Also sometimes in medical imaging the conditions are very different from actual practice. For example the doctor may be worse than the convnet on certain types of low-quality, low-dynamic range images that someone preprocessed in a particular way. But sure, in the medical field some error prone, boring counting tasks and spot-the-cancer-in-your 200th-image-today, the machine can perform actually better.

But what tasks specifically do you have in mind?

That is a rather contentious claim.

>Understand context across a whole book.

to have truly much utility when it comes to context a model doesn't just need to correlate information across some text, it also needs general knowledge and understanding so that it can produce knowledge which is only implicit or not even present in the text itself.

You can make the transformers as large as you want, NLP models still fundamentally suck at answer trivial questions like "If Pete was alive in 2000 and alive in 2050, was he alive in 2025?"