Re:Turing-NLG: A 17B-parameter language model
by corporateslave5 on Monday February 10, 2020 @03:53PM
(#22293169)
Maybe. I take a really optimistic view of attention based mechanisms, like I explained and added to my original post. If you read the recent reformer paper, they produce a new way of computation to start building a model that can in some way, encode the relationships between different parts of war and piece. The bottle neck right now is computation, we don’t know how well these models can learn when that bottle neck is removed!
I’m optimistic because I believe that the contextual information that you are describing, is already there in the vast expanse of the internet.
But I will also add, I think none of this will spawn AI, just that it will spawn new technologies that are categorically different.
Re:Turing-NLG: A 17B-parameter language model
by m0zg on Monday February 10, 2020 @04:50PM
(#22293774)
I think it will merely spawn technologies that are less fragile, yet still too fragile and unsophisticated to be practical for full blown conversational user interface the likes of which you see in the Avengers movies. It will (and already does) make a difference in simpler tasks where you can get away with just counting numbers, such as search/retrieval, simple summarization, constrained dialogue in chatbots, stuff like that.
Let's take a question answering example. Take just about any recent deep learning paper and try to answer detailed, higher level questions against it. To use a concrete example, take MobileNet V3 paper and ask your system "do I put activation before or after squeeze and excitation" (correct answer is "before"), or "do I need a bias in squeeze and excitation module" (correct answer is "it depends on the task"). You won't be able to, because a lot of things are just _assumed_, just like in any other realistic example of text written for human consumption. The facts are encoded externally as information about the world, and they're so fine grained and contextual, that we don't even know how to begin incorporating them into the answers, let alone do so contextually and probabilistically, like human mind does.