I take the point to be that there aren't "deeper" fundamental principles at play in these models. Tremendous progress has comes from simply tweaking of the numbers of layers, or how the feed forward to each other (skipping layers, etc), or by throwing more computer power or data at the same basic algorithm.

Where might we look for deeper principles? One idea is to consider what brains do and how they might be doing it. (I'm not saying we need to go down the rabbit hole of biological detail -- on the contrary I'm suggesting we look at known or even hypothesized principles of brain operation and import them into AI.)

Two ideas we have used in our work: prediction (over time), recurrent feedback (most brain regions have more feedback than feedforward inputs)

As a physicist judging from the outside, I share some your feeling. Are there general laws governing "learning"? Theorems? Are there "deeper" things to learn as humans? The thing is people in the field don't need heavy intuition or math. In some ways that's good (if you just want a result to utilize) and in others, it's bad (if you are a curious person).

In a sense, I would say yes there are learning laws, but it's still early in codifying them.

Along one axis, you could compare: supervised, semi-supervised, self-supervised and unsupervised learning. Along another axis, consider that there are versions of each method that take into account temporal/dynamic data, versus others that require randomly shuffled static data.

In the current problems of visual perception, I think the field would benefit greatly a shift to focus on multiscale interaction/dynamics rather than on (static) statistics as it is currently (for more on this, see my colleague's blog: [1]).

[1] Statistics and dynamics. (http://blog.piekniewski.info/2016/11/01/statistics-and-dynam...)

>In the current problems of visual perception, I think the field would benefit greatly a shift to focus on multiscale interaction/dynamics rather than on (static) statistics as it is currently (for more on this, see my colleague's blog: [1]).

Your friend's blog has a lot of good insights that I've seen in the theoretical neuroscience and computational cognitive science literature as well. Where do you guys work?

I work at LeEco US out of San Diego, and my colleagues work at other ML/AI companies also in San Diego. We originally met and collaborated at Brain Corporation.

Are you thinking of laws along the lines of biological evolution or physical laws of motion, where very simple ideas can produce extremely complex emergent behavior?

If so I think a layman law of ML is already understood.

To draw an analogy, some programmers might use trial-and-error to produce programs by fiddling with a few lines of code, seeing if it pass more unit tests, repeat.

If you believe that a human brain can be represented by machine code, then given infinite time, that trial-and-error programmer can write down the "source code" of the brain.

Then machine learning is just a "turing complete programming language"(i.e., a neural network architecture) with "source code" (in the form of matrix weights) where "passing more unit tests" is done by numerically following a gradient to update the "source code".

Everything else is just finding a better "programming language" that can make this run very fast on our current machines.

> Are there general laws governing "learning"?

This has been my question too...

Based on my understanding, it all really boils down to probability, statistics and a few important theories like Vapnik-Chervonenkis which provides the mathematical foundation for what "learning" is, can we even learn from the given data and how well can we learn (VC dimension, etc).

But I would love it if someone can point me to or explain / derive from core first principles the concept of "learning".

This is what the experiment does. The theory they are looking for/validating/falsifying is much deeper and richer than just a multi-linear fit.