News for hackers. Stuff that matters.
Re:AlphaGo Zero: Learning from scratch by etiam on Wednesday October 18, 2017 @04:20PM (#15502584)
Please don't refer to them as black boxes. The internals are fully observable.
Re:AlphaGo Zero: Learning from scratch by andbberger on Wednesday October 18, 2017 @08:46PM (#15504574)
Fully observable and we still have no idea what the hell it's doing.

Makes neuroscience seem kinda bleak doesn't it?

There has been a lot of great work lately building up a theory of how these things work, but it is very much still in the early stage. Jascha Sohl-Dickstein in particular has been doing some great work on this.

We don't even have answers to the most basic questions.

For instance (pedagogically), how the hell is it possible to train these things at all? They have ridiculously non-convex loss landscapes and we optimize in the dumbest conceivable way, first-order stochastic gradient descent. This should not work. But it does, all too often.

Not a great example because there are easy hand wavy arguments as to why it should work, but as far as proofs go...

The hand wavy argument goes as follows: - we're in like a 10000 dimensional space, for the stationary point we're at to be a true local minima that means each one of those 10000 dimensions goes uphill in either direction. It's overwhelming likely that there's at least one way out - there are many many different ways to set the params of the net for each function. Permutation is a simple example.

We really have no idea how these things work.

Anyone who tells you otherwise is lying to you...

Re:AlphaGo Zero: Learning from scratch by otabdeveloper1 on Friday October 20, 2017 @05:14AM (#15514486)
> Fully observable and we still have no idea what the hell it's doing.

Of course we do. It's matching a smooth multi-dimensional curve to sample data.

Re:AlphaGo Zero: Learning from scratch by fragsworth on Wednesday October 18, 2017 @04:27PM (#15502677)
It's a conceptual black box. There's no way for us to understand what each individual neuron is doing.
Re:AlphaGo Zero: Learning from scratch by blt on Wednesday October 18, 2017 @06:22PM (#15503769)
The tools we have developed so far are limited, but that's different from "there's no way". Many academics are working hard right now to better understand deep neural networks.
Re:AlphaGo Zero: Learning from scratch by etiam on Wednesday October 18, 2017 @04:39PM (#15502789)
Re:AlphaGo Zero: Learning from scratch by ionforce on Wednesday October 18, 2017 @04:28PM (#15502692)
Is there meaningful information in what one observes?
Re:AlphaGo Zero: Learning from scratch by aeleos on Wednesday October 18, 2017 @04:59PM (#15503024)
Yes, it turns out you can find meaningful information. etiam provided this https://arxiv.org/pdf/1312.6034.pdf The main issue is making sure what you are looking for is actually what the network is doing. You have to correctly interpret and visualize a jumble of numbers, which usually requires a hypothesis about how it worked in the first place. But assuming both go well you can gain meaningful information.
Re:AlphaGo Zero: Learning from scratch by fjsolwmv on Friday October 20, 2017 @03:49PM (#15518673)
Can I train an NN to visualize the numbers?