News for hackers. Stuff that matters.
Re:Geoffrey Hinton spent 30 years on an idea many other scie... by taeric on Saturday February 03, 2018 @02:10PM (#16299007)
This is an odd story that seems to gloss over the downsides of neural networks. Computational power needed to build some of the models is enough to explain how slow uptake was. At least in large. I would be interested to see just how many multiplications go into a typical model nowadays. In particular, the training of one.

But that still skirts the big issue, which is generalization. We are moving, it seems, to transfer learning. The danger is that we don't seem to have a good theory to why it works. At a practitioner level, I don't think this is as much of a problem. For the research, though, it is pretty shaky.

I think there is more than a strong chance this remains the future for a while. And I am layman in this field. At best. But this story presupposes that the past was wrong for not being like the present. That is a tough bar.

Re:Geoffrey Hinton spent 30 years on an idea many other scie... by js8 on Saturday February 03, 2018 @03:25PM (#16299432)
I believe neural networks are not quite the right approach, and their success is misleading.

Historically, almost all ML approaches were based on separation in Euclidean (vector) space, which is understandable because they were developed for much weaker computers. However, really useful ML tasks require to deal with huge nonlinearities, and the fact that you're training in linear space becomes less relevant.

Neural networks have surmounted the nonlinear difficulty by increasing the number of layers. But it's a question whether a similar result couldn't be achieved with Bayesian networks on binary representations (which is approach I favor).

There is some evidence that the precision of linear calculation in neuron (for example, resolution of the weights) doesn't really make much difference in the performance of neural network. Could it be that the neural learning through vector manipulation is just an artifact of the origin of the neural networks, and the really important thing is the overall organization of the network (the layers)?

Re:Geoffrey Hinton spent 30 years on an idea many other scie... by taeric on Saturday February 03, 2018 @03:59PM (#16299621)
I'm assuming people are trying with Bayesian methods. Any good research it practical examples showing advantages?
Re:Geoffrey Hinton spent 30 years on an idea many other scie... by relate on Saturday February 03, 2018 @02:43PM (#16299172)
I find it strange you mention transfer learning, since one of the reasons neural networks are so popular are because they tend to excel at it. Adapting (i.e. fine-tuning) networks trained on a task with a lot of data (e.g. image classification on ImageNet) to different tasks, such as image segmentation, has proven a very successful approach.
Re:Geoffrey Hinton spent 30 years on an idea many other scie... by taeric on Saturday February 03, 2018 @03:56PM (#16299606)
That was my intent in mentioning them. My understanding, though, is that we don't have a solid foundation on why it works. Just that that is the direction we are moving, and neural networks are good at it.
Re:Geoffrey Hinton spent 30 years on an idea many other scie... by seanmcdirmid on Saturday February 03, 2018 @02:33PM (#16299116)
Researchers are supposed to be looking toward the future, and shouldn't be hindered by the limitations of the present. So even if your methods are not feasible today, you can still investigate them and be ready for when they will work. That many of those other scientists didn't have the foresight to see that this would work someday is definitely a big miss on their part. Let's not give them a pass.
Re:Geoffrey Hinton spent 30 years on an idea many other scie... by taeric on Saturday February 03, 2018 @03:58PM (#16299614)
That is somewhat silly. Especially when you can't investigate using the tools of the day.

Be clear on that. He was not able to see the success using these techniques on older hardware. Period. People weren't dismissing him. They were doing better.