News for hackers. Stuff that matters.
Re:Failures of Deep Learning by curuinor on Friday March 24, 2017 @06:29PM (#13952738)
Why can't we pull out the statistical mechanics? You can't understand the 10^27 variables in a lump of coal, either, but you can still do thermodynamics. People already do it, Ganguli, some stuff from Bengio
Re:Failures of Deep Learning by bitL on Friday March 24, 2017 @07:16PM (#13953014)
I wish I were so optimistic - if you use statistics, then you might remove most of interesting outliers and study just those more stable prevailing characteristics and not the problematic ones, like when using Hooke's law for designing buildings while ignoring its limited scope. And physics hits hard math limits while studying dynamic systems (i.e. fractals) anyway.

We can always use Ramsey Theory stating there is no chaos possible and you can always find some regularity in anything, yet if you need to scan 10^132 items to find the order it becomes quickly uninteresting.

Re:Failures of Deep Learning by curuinor on Friday March 24, 2017 @07:22PM (#13953046)
RNN is natively and obviously dynamical but FFNN can also be seen as transient dynamical system.

Little Ramsey theory ever used in neural network land, but theory of dynamical systems used from quite a long time ago, from Werbos-BPTT which was explicitly envisioned as a dynamical algorithm to the description of the dynamical instability of gradients in RNN.

The statistics of statistical mechanics don't necessarily need to be CLTish sorts of things, you know?

Re:Failures of Deep Learning by bitL on Friday March 24, 2017 @07:30PM (#13953092)
Possibly ;-) I'll study these things in detail soon (hopefully), so far just practical experience with all funny things from DL like self-driving cars, composing music with RNNs etc. but having extensive operations research exp in the past I tend to be careful.