You might find this 1996 Nature paper by Olshausen & Field interesting. In it, they describe how a coding strategy that maximizes spareness when representing natural scenes is enough to produce a family of localized, oriented, bandpass receptive fields, like those found in the early visual system of humans.

https://courses.cs.washington.edu/courses/cse528/11sp/Olshau...

This fits in with my point: they imposed several restrictions about the model space: maximizing sparseness, and they also make several linearity assumptions.

Superficially, those graphics look a lot like visualizations of conv nets.

those papers should be, if not 101, at least 201 for neural net track as it would help to establish common framework and basis for thinking, talking and analysis of neural nets machinery.

Great, now I've got another paper to cite. Thanks!

>you can argue that Gabor filters arise because we design neural nets to encourage them. Gabor filters mostly arise in CNN or things otherwise regularized to be like CNNs.

and do we know why? Usually it would mean some optimality. It should be relatively simple math here (back at the time at our University it would be given to a student as a thesis project and couple months later we'd have it), and that would give us 2 things - insight into biological visual cortex (which we suppose follows some optimality too and know we would have a very good candidate for the one) as well as to allow to start some primary convolutional layers with the (optimal set of) Gabors instead of going through learning them. Actually some of the best results i saw 15-20 years ago were produced by the simulation of visual cortex through such construction. And now image trained deep learning nets converge to the same.

No, we don't know why. You also have little basis to claim the biological visual cortex is optimal. It certainly works, which is enough for people who draw inspiration from it.

that is my point - we have very good suspicion and experimental data (successful deep learning CNN as well as visual cortex) that it is optimal, at least in some very wide class if not global, and that warrants investigation for a proof of it. Proven optimality would be very telling, especially for biological visual cortex. Even if optimality doesn't happen for CNN Gabor, there may happen to be discovered reasons for why not, and thus that would help to construct even better, probably optimal, approach.

Suspicion and empirical evidence cannot prove something is optimal. You cannot exhaustively empirically search an infinite space of models. You are seriously misunderstanding the definition of "optimal."

>Suspicion and empirical evidence cannot prove something is optimal.

what are you talking about? where did i say such a thing?

> we have very good suspicion and experimental data ... that it is optimal

...

I am not seeing the chicken and egg problem. Isn't it always the case that when we consider the optimality of something it depends on some definitions?

The chicken and egg problem is that we design neural nets in order for Gabor filters to show up. If you used a different neural net architecture choice, they wouldn't show up. So the presence of Gabor filters indicating optimality is sort of begging the question.

As far as I know, Gabor filters show up in all kinds of multi-layered models, even in unsupervised ones. Gabor filters are quite canonical.

CNN beat other architectures in images and Gabor emerges in CNN. That is a pretty good candidate for even global optimaliy.

The fact that X works and A, B, and C don't work, does not imply A is globally optimal.