Quoting from the Universal approximation theory's Wikipedia, "neural networks can represent a wide variety of interesting functions." While they may be much better at pattern recognition, it's possible to produce almost anything with one, including Go moves, if you can devise a method to interpret the outputs.
I think that's "true and false". Yes, NNs produce a "chance of being X" but no, NNs don't naturally produce a human-intuitive "degree of being X". If the NN is looking for a "red fire engine", it's not necessarily going to produce "degree of redness" as its output.
-- As to neural nets representing many functions; neural nets, Taylor series, Fourier series and so-forth can represent/approximate "any function". True but effectively irrelevant. What matters is what function a given methodology can be trained, programmed or whatever into representing.
That's incorrect. https://arxiv.org/abs/1602.02830
Descartes Labs uses the pre-trained ResNet 50 and removes the final layer which does classification. What's left is a layer that provides image features necessary to do classification. These features can be used to sort images by similarity and search for similar images.
They get a vector of approximate features and can use it match to other images.
BUT there's still the "this means nothing" problem. The vectors, as far I can tell and by the logic of just doing autoencoding, don't have a significance except for the system. Can find image X and say it's like image Y.
But it doesn't help at all at finding specified things. You can't say "find me a corn field" or "find my nuclear power plant". You can show it a picture of nuclear power plant and it can show you mountains with a similar layout.
This is one thing I worry about, that even if we were to get closer to AGI, the algorithms/system might not feasibly run with our current computing paradigm.
We have human beings who seem to function normally in situation where their brains have actually shrunk to a fraction of their skull size. And we animals that can do things robots can't do but will very small brains indeed. So something like an intelligent algorithm seems possible with "small" "hardware".
Oppositely, it seems like hypothetically one could pretty easily specify an "intelligent algorithm" if one had a computer of limitless speed and memory.
So approaching AGI would seem to involve producing things of sufficient efficiency and there's reasons to think that's possible.
Which is to say that convolutional neural networks, the most common and successful neural networks, fundamentally only operate by recognizing a broad category (face, nose-on-face, mouth-on-face). This limitation seems as inherent in NNs operating with a black-box/training-set/testing-set cycle as it is in any architecture consideration. If all you can "tell" the NN is "this is X, this is not X", then it's plausible that's all it will give back. Moreover, "X/not-X" testing allows a LOT of training to happen in an unambiguous fashion, which seems necessary when what you're doing gradually pushing a giant bunch of semi-arbitrary detectors into place.
Machine learning theorists can do a lot with nothing but this category-recognition tool - like having a go program recognize "good moves" and "good positions". Note, that the more elaborate "neural net within neural net" approaches involve vast levels of computing power - you're brute-forcing brute-force. AlphaGoZero probably involved the largest level of computing power ever harnessed for a single-use app. AlphaGoZero involve megaflop levels close to those estimated for the human brain though such estimates are always debatable (perhaps why we didn't see many games by it).
The thing is there are plenty of ordinary computing algorithms do "more" than neural nets - perform logical operations or sort and search by vague similarity or classify on spectrum of multiple qualities (rather than the binary X or not-X of a neural network). It is just that these break down at an "industrial scale", whereas neural nets are shine at this scale.
However, despite these virtues, it seems likely we'll see neural nets approaching their particular limits reasonably soon, if they haven't (in at least a sense) done so already.
And it seems like any "real" extension/alternative will have to be able to use big data but use it in a more sophisticated fashion than neural nets.
Oh, this relates to the video most in that when Bengio talks about how to extend neural networks, he's mostly just mentioning extension in terms of how to add abstract qualities like "causation", where one can certainly see crude, simplistic things in computation that could be called "working with causation" but you don't have that yet when you go to the level of a large-scale neural network.