News for hackers. Stuff that matters.
Re:A visual proof that neural nets can compute any function by akuma73 on Tuesday September 02, 2014 @02:12PM (#8259168)
What about XOR?

From Wikipedia: In 1969 in a famous monograph entitled Perceptrons, Marvin Minsky and Seymour Papert showed that it was impossible for a single-layer perceptron network to learn an XOR function.

Re:A visual proof that neural nets can compute any function by jbarrow on Tuesday September 02, 2014 @02:21PM (#8259222)
The simple answer is that we no longer use the standard linear perceptrons that McCulloch and Pitts put forth in 1943. The limitations of perceptrons and their training algorithms are well-known and well-studied, and in the last 60 years we've come a long way.

A book came out a little bit ago which covers basic neural networks, and culminates in showing a network that can estimate XOR [1]. I strongly recommend reading not only the chapter, but the whole book.

[1] http://natureofcode.com/book/chapter-10-neural-networks/

Re:A visual proof that neural nets can compute any function by jo_ on Tuesday September 02, 2014 @03:21PM (#8259549)
'Single-layer' is an easily overlooked and absolutely essential modifier. Very few networks these days are single layers. (Three is the minimum for infinite dimensional functional approximation, given nonlinear activations.) Deep Networks you see in new research papers these days have, at minimum, perhaps three layers. Some by LeCun et al. will go as high as nine or twelve layers, with four or sixteen layers in breadth.
Re:A visual proof that neural nets can compute any function by szabba on Wednesday September 03, 2014 @03:08PM (#8264694)
I'm pretty sure linearity was also part of the picture -- no matter how many layers of neurons you have, as long as they perform linear combinations, they could all be replaced with a single one (ignoring numeric errors).
Re:A visual proof that neural nets can compute any function by jo_ on Wednesday September 03, 2014 @03:20PM (#8264766)
I mentioned this in my post ("given nonlinear activations"), but you're correct to point out its importance. The combination of linear functions is always linear. Full stop. You need at least one non-linear layer to get reasonable results.
Re:A visual proof that neural nets can compute any function by Dn_Ab on Tuesday September 02, 2014 @02:19PM (#8259208)
If you join a bunch of perceptrons together that limitation goes away. Another path is to make the problem effectively linear again by transforming into higher dimensions, kernels do this with one clever trick that allows them to avoid the computational cost of doing so explicitly.
Re:A visual proof that neural nets can compute any function by JadeNB on Tuesday September 02, 2014 @03:42PM (#8259667)
> kernels do this with one clever trick that allows them to avoid the computational cost of doing so explicitly

Minsky and Papert hate him! Use this one weird trick ….

(Sorry; I couldn't resist!)

Re:A visual proof that neural nets can compute any function by gwern on Tuesday September 02, 2014 @04:28PM (#8259956)
Sorry, you were beaten to the joke a long time ago: http://www.oneweirdkerneltrick.com/
Re:A visual proof that neural nets can compute any function by JoeAltmaier on Tuesday September 02, 2014 @02:14PM (#8259176)
Neural nets now use non-linear functions so they can do things like xor.
Re:A visual proof that neural nets can compute any function by dragonwriter on Tuesday September 02, 2014 @02:22PM (#8259226)
"Single-layer" is an important restriction there, however.
Re:A visual proof that neural nets can compute any function by pesenti on Tuesday September 02, 2014 @02:18PM (#8259203)