Geoffrey Hinton and Jürgen Schmidhuber are convinced that neural nets trained with backprop will likely lead to AI with minimal additional fixed-function structure. Schmidhuber's intuition is that recurrent neural networks (RNNs) are capable of universal computation, so given sufficient computational resources the RNN can learn any task, including generalization, goal and action selection and everything else that we associate with intelligence. Hinton's intuition is that RNNs essentially accumulate representations in their hidden state vectors that are very similar to human thoughts ("thought vectors"). Hinton thinks that thought vectors will naturally lead to the kind of reasoning that humans are capable of. In his view the human brain is basically a reinforcement-modulated recurrent network of stacks of autoencoders (needed for unsupervised representation learning).
Pedro Domingo sees such universally that may lead to AI in all major fields of AI: (1) Symbolists have inverse deduction (finding a general rule for a set of observations), (2) Connectionists have backprop and RNNs (see above) (3) Evolutionists have genetic programming (improvement of programs via selection, mutation and cross -over), (4) Bayesianists have incremental integration of evidence using Bayes' theorem, e.g. dynamic Bayes networks, and finally (5) Analogizers have kernel machines and algorithms to compare things and create new concepts.
Domingo's hypotheses is basically that only a combination of various of these approaches will lead to AI.
If the Strong Free Will theorem is right, there are distinct limits on the ability of computational devices to simulate our physics.
For one, anything a computable machine does "on its own" will be a function of the past history of the universe (deterministic), whereas the SFWT says that elementary particles are free to act without regard to the entirity of history.
E: even a one word reply conveys so much more than an anonymous downmod.
Furthermore a ai does not compute a physics model any more than a brain does, so the criticism does not apply.
The point is that the universe can do things that functions cannot. If your only tool is functions (only thing that algorithms can compute), then you will necessarily be unable to handle all possibilities (by which point, I question your intelligence).
But even so, it's confusing to phrase intelligence in terms of non-determinism. It's easy to come up with a non-deterministic answer to an arbitrary question. It's hard to come up with a correct answer to an arbitrary question. If unpredictability is a component of sound reasoning, it's because we humans are so bad at reasoning.
The problem is not that, "my answer must be 'unpredictable'", but, "the actual answer may not be computable" (and so, no algorithm may ever derive it).
Dualism is a very lonely position around HN. You and I may be the only dualists here.
> "Our argument combines the well-known consequence of relativity theory, that the time order of space-like separated events is not absolute, with the EPR paradox discovered by Einstein, Podolsky, and Rosen in 1935, and the Kochen-Specker Paradox of 1967"
So as far as I can tell, it takes for granted the humans' ability to choose the configurations freely, which though suspect in of itself doesn't matter so much to their argument as it's not really an argument for free will, it's a discussion of how inherent to quantum mechanics non-determinism is.
> "To be precise, we mean that the choice an experimenter makes is not a function of the past."
> "We have supposed that the experimenters’ choices of directions from the Peres configuration are totally free and independent."
> "It is the experimenters’ free will that allows the free and independent choices of x, y, z, and w ."
It is actually, if anything, in favor of no distinction between humans and computers (more precisely, it is not dependent on humans, only a "free chooser") as they argue that though the humans can be replaced by pseudo random number generators, the generators need to be chosen by something with "free choice" so as to escape objections by pendants that the PRNG's path was set at the beginning of time.
> The humans who choose x, y, z, and w may of course be replaced by a computer program containing a pseudo-random number generator.
> "However, as we remark in [1], free will would still be needed to choose the random number generator, since a determined determinist could maintain that this choice was fixed from the dawn of time."
There is nothing whatsoever in the paper that stops an AI from having whatever ability to choose freely humans have. The way you're using determinism is more akin to precision and reliability—the human brain has tolerances but it too requires some amount of reliability to function correctly, even if not as much as computers do. In performing its tasks, though the brain is tolerant to noise and stochasticity, it still requires that those tasks happen in a very specific way. Asides, the paper is not an argument for randomness or stochasticity.
> ” In the present state of knowledge, it is certainly beyond our capabilities to understand the connection between the free decisions of particles and humans, but the free will of neither of these is accounted for by mere randomness."
>There is nothing whatsoever in the paper that stops an AI from having whatever ability to choose freely humans have.
There is if an AI is dependent on deterministic methods. I agree that AI is not a well-defined term, but all proposals I have seen are algorithms, which are entirely deterministic. This is entirely at odds with the human conception of free choice. An algorithm will always produce the same choice given the same input. Any other behavior is an error.
The SFWT says that observations can be made that cannot be replicated through deterministic means, which would seem (I agree there is a very slight leap in logic here) to rule out any AI from ever being able to understand at least some aspects of our reality (and also reveals them to be simple, logical machines, with no choice).
Computer vision is such a field. The results provided by conventional approaches had been plateauing for years and then one day, Yann Le Cunn submitted a paper describing the work that he and his team had done in the area with neural networks and which blew away the previous results on image recognition.
Interestingly, this paper was rejected and it took a full year to the CV community to finally turn around and accept NN as a valid approach to this problem. I believe no one questions that it's the best methodology we have today.
Speech recognition, automatic translation and natural language processing in general are other areas that have benefited immensely from neural networks.
When I read content about 'deep learning' neural nets today I don't see anything especially different to what my (admittedly shallow) understanding of neural nets was back then. So what I'm missing mainly is - what changed? Is it just that advances in compute power mean that problems for which neural nets were impractical have now become practical? Is there something different about the way neural nets are employed in 'deep learning' that is different than the neural nets that were discussed in the past?
Deep learning got started when Hinton observed that a certain way of training restricted Boltzmann machines wouldn't get stuck as easily, and hence by pretraining the network as if it were an RBM and then switching to backprop, it wouldn't get stuck in a local minimum as early.
As I understand it, nowadays the best method looks something like a generalization of Newton's Method, wherein the direction you move takes into account the second differential and not just the first differential or direction of sharpest descent. You move furthest in the directions that curve the least, and move the least in the directions that are most sharply curved. It turns out that this (plus some other tricks) make it way easier to follow continuous gradients in big weird parameter spaces, so now it's possible to train deep nets, which are a kind of continuous gradient in a big weird parameter space.
Tl/dr: People figured out how to move better through the parameter space of neural networks, by taking into account the second differential plus some other tricks. So now we can train deeper nets.
[0] https://en.wikipedia.org/wiki/History_of_supercomputing#Hist...
The rectified linear unit also converges about 75% faster than the classic Sigmoid or Tanh. the ReLu is F(x) = max(0, x) so the gradient propogated is 1 rather than a more "curvy" value.
In NLP in particular, I'm not so sure about the "immense" benefit of ANNs. LSTM-RNNs are very nice for language modelling but we haven't seen the 10-percentile jump in performance that the image processing community has. Stanford I believe uses ANNs in their dependency parser (to learn transitions) and Google of course uses ANNs everywhere, but the gains against goold old HMMs are nowhere near the gains in image processing and it's not clear what exactly is the difference. Maybe teams have better hardware and more data now. Rather, they definitely do, so maybe that's the reason for any increases in performance.
With translation in particular (it's a branch of NLP) it's very hard to know what's going on. Companies who sell machine translation products still use different techniques, including rule-based software. The problem is that machine translation needs some way to deal with semantics, else it's a bit shit. Google again- they do use ANNs (apparently- it's hard to tell because they don't) but is their translation any better than others'? That's almost impossible to know, because there are no good measures of machine translation performance. I'll repeat that: we don't have any good measures of machine translation performance. So anyone can come up with a solution and drum it up as "state-of-the-art", and if their user base is large enough it'll be impossible for anyone (even themselves) to know for sure whether that means anything.
Speech recognition- I don't know much about that, but I think it's similar to NLP. You can do a lot with a lot of data and hardware, if you're lucky to have them and at that point it doesn't really make a real difference to your end-user if you're using an ANN model or an HMM model.
Which may just go to show that Hinton, Schmidhuber and the rest of the connectionist gang are right about the value of more data and faster computers.
https://plus.google.com/+YannLeCunPhD/posts/gurGyczzsJ7
I seem to remember this article as having a lot of very interesting comments, including from the chairman of the conference. The exchange was cordial but showed some clear antagonism between the two parties. Somehow, I don't see these comments any more on Google+.
That was four years ago, things are a lot better now.
I believe you'll be hard pressed to find anyone that describes expert systems as "learning" in any way. They're typically considered as the antithesis of machine learning: they're knowledge-driven but you have to enter all the knowledge "by hand". Indeed some machine learning algorithms were proposed as ways to automate the creation of the large rule-bases typical of expert systems (the most prominent such algorithm being Decision Trees).
>> It would be interesting to hear the thoughts of a non-NN AI researcher regarding this.
Researchers are rarely as proscriptive as that. You may find folks who specialise in ANNs to the exclusion of other techniques (but it's true it's a vast field) but you'll probably not find anyone who wouldn't try ANNs a couple of times, and maybe publish a paper or two on them. Same goes for other trendy algorithms also.
There are researchers actively looking at more holistic AI systems that aren't just the core learning part (often called cognitive architectures). For instance, take a look at Soar: http://soar.eecs.umich.edu/
It under active development, and has been since the 80s. Soar is very much a spiritual successor to GOFAI (or maybe still is GOFAI), but does incorporate statistical machine learning-- much of the recent work has been around integrating reinforcement learning with symbolic decision-making.
ACT-R is another cognitive architecture under active development, but is more used for cognitive modeling (i.e. psychology research): http://act-r.psy.cmu.edu/
You can find other related systems by searching for "cognitive architecture", but it is unfortunately a field that attracts a lot of proposals/'designs' for things that never get properly implemented (the above being two notable exceptions).
http://www.springer.com/us/book/9783540200406
Lise Getoor from MIT is also doing a lot of work with statistical relational learning in particular, which is kind of a cross between GOFAI (the "relational" bit) and nouveau AI (the "statistical" part):
http://www.cs.umd.edu/srl-book/
In the UK I know of Kristian Kersting who does statistical relational learning research, at Dortmund:
http://www-ai.cs.uni-dortmund.de/PERSONAL/kersting.html
I think this is not exactly what you were asking for but I think you might find it interesting, particularly Inductive Logic Programming (of which there's oodles in De Raedt's book).
Personally, I started an MSc in AI because I was interested in all the symbolic stuff, and I was a bit disappointed to find there were almost none of that left in the curriculum (at my school anyway, the University of Sussex). I guess I kind of see the point though- knowledge engineering is pretty hard and costly and learning from data is very attractive. I really don't like the idea of doing it all with statistical black-boxes, though. So relational learning is a kind of halfway house for me, where I can still use mathematical logic principles and not have to eat the statisticians' dust, so to speak. For me relational learning is the spiritual successor of GOFAI that I was looking for, so I think you may find something of interest in the links above.
Edit: there's also the probabilistic logic programming community, forgot about them; here:
https://dtai.cs.kuleuven.be/problog/
That's from the University of Leuven, where De Raedt is from. The name says it all: Horn clauses with probabilistic weighting. So that's not machine learning (or any kind of learning, it's just what it says on the tin). You will find many more here:
http://probabilistic-programming.org/wiki/Home
Although that's not all _logic_ programming (but there's quite a bit of it).
I'm thinking of evolutionary algorithms, various other biologically inspired computation techniques (of which NN's are but one example), more traditional AI techniques such as expert systems, and a whole host of stochastic, non-biologically inspired algorithms.
I'm not super familiar with NN's myself, so I can't say whether the gigantically disproportionate attention from the media and the research community is deserved based on actual superiority in effectiveness of NN's compared to other techniques, or whether they're used and talked about mostly because that's what most people know.
It would be interesting to hear the thoughts of a non-NN AI researcher regarding this.