It's currently getting the best results, which isn't exactly the same.

Deep learning undeniably works well. It works so well that it has almost completely taken over the field, sucking the oxygen out of other parts of the field. This is a little disconcerting since no one really understands how much of deep learning's success is due to a) the algorithms themselves vs. b) the massive amount of data and computational power being thrown at vision problems.

There is something to be said for some "counter-cyclical" efforts to encourage people to keep exploring approaches that aren't just slightly different network topologies, or least to ensure that they're not totally forgotten. That's the message I took away from the article.

For a time neural networks seemed dead and SVMs were sucking the oxygen out of other parts of the field. Then someone figured out how to train deep networks and the tides turned.

Sure, though the SVM tide never submerged competing methods quite as deeply as the incoming deep network tide, either in terms of performance or mind-share. Ensemble methods also became popular around the same time, for example.

In a way, you could take this as a perfect example of the authors' point: we might have had a lot of these advances sooner if we hadn't let interest in neural networks fall completely off a cliff.

Sure, if we had unlimited money and could pay people to work on stuff that doesn't seem super promising, we could have had a lot of things sooner.

We have unlimited money. But only when banks or the defense sector needed it. That's not a rant, that's an observation.

Think about it: Do we really, in the US of A or anywhere in the 1st world countries, have a lack of resources? In the 21st century? With hardly anyone actually working on the basics the population needs, like food, because our productivity is through the roof?

And "money" only is an issue when humans decide it is - it is an entirely made-up concept represented only in bits in computers (never mind that little bit of printed money, which can be produced at will too).

We do pay a huge amount of people in both private and public sector to do useless work. See dilbert.com cartoons and their popularity, and the results of research into the mood of working people, a huge amount of them thinking their work is useless to society or even just their own company.

The feeling (and reality) that we all have to work hard to get by is mostly an artificially created system outcome. With the productivity levels we have achieved, and the tools and knowledge at our disposal, we should have plenty of room for experiments.

> Think about it: Do we really, in the US of A or anywhere in the 1st world countries, have a lack of resources? In the 21st century?

Yes, I just say national debt. As long as this is not paid back (or we are working seriously to decrease it) we can't claim that we have an abundance of money - quite the opposite. Even more: As long as we aren't seriously working to decrease the national debt we are near to delayed filing of insolvency, which is not a good idea as any founder can tell you.

National debt? Really? You know that the modern money system IS debt? No debt - no money.

And no, debt does NOT have to be paid back. It can be rolled over indefinitely, or, if demands on "payments" really are an issue just go bankrupt. NOT A JOKE. It happened A LOT. I myself - and I'm not that old - have seen three currencies in my country in my life (Germany), my grandparents more. Germany is a horrible 3rd world place... no?

What are people going to do when all they get is "sorry, no more payments on the old debt. Are they going to emigrate to Alpha Centauri? Are they going to refuse to live from now on? No, they will grumble and get back to doing what they've been doing all the time. As long as the government is strong etc. Plenty of examples - see Germany. No that doesn't lead to Argentina - only Argentina leads to Argentina. A countries industry, culture, people, government don't suddenly stop working just because the currency is changed and old debts forgiven. Money does not exist, it's a made-up concept.

> we can't claim that we have an abundance of money - quite the opposite.

Uhmm... what should I tell someone like you? Except to repeat:

Money does not exist, it's a made-up concept.

Guess what happened after the financial crisis - we invented a trillion dollars. Out of nowhere. Suddenly it was there.

I don't think this an adequate analogy. SVMs never completely captured the field. There were also random forest, boosting and other promising algorithms that delivered results "in parallel" (within the same time-frame).

Also, the attitude that ANNs are "all we need" was there even before deep learning delivered all the recent SOTA results. I remember Patrick Winston commenting on that in one of his lectures in 2010(!).

By "how to train deep networks" are you referring to unsupervised pre-training, or the discovery/development of faster parallel training? (GPUs and distributed ASGD)

It gets the best results for what? Maybe for classification and recognition.

But the point of the article is that there is more to computer vision. Stereo, optical flow, geometry, and physics can only be aided by deep learning so much.

Another point not mentioned is the computational power required for deep learning. Consider programming the physics for a ball rolling down an incline. You could use (1) the math itself, vs. (2) a neural net. It's clear that the direct math approach could be orders of magnitude faster than neural nets. I wouldn't be surprised if directly coding the physics would achieve 1,000x the performance of a neural net.

I would posit that a deep learning network that learns to optimize parameters for a complex algorithm outside the convolutional network itself may have immense utility outside the classification problem. Call it a marriage of classic computer vision with deep CNN, or a hybrid approach. I don't think it's a binary decision. A deep CNN can find the optimal parameters (once trained) for a classic CV problem for a given image or video or other dataset, like superresolution, patch-based inpainting, or motion tracking. The training is the most computationally intensive part. As someone with way too many kids, I can testify...

You seem to be describing hyperparameter optimization, which I seriously doubt CNNs are going to be used for anytime soon.

It has been done already: http://arxiv.org/abs/1502.05700

Edit: Although what they seem to describe is replacing GPs with neural networks in Bayesian optimization which is supposedly more efficient.

Since the point of Bayesian optimization is to limit the number of times you have to evaluate a new set of hyperparameters, I am not sure how useful it is to "be able to scale" (i.e. even if maintaining the GP is O(n^3) with the number of evaluations, the costly part should be to evaluate the hyperparameters in the first place) but I haven't read the paper so they may show some high dimensional hyperparameter cases where performing a lot of evaluations pays off.

Just curious -- why would you think not?

The point of all of those though are toward the goal of machine/computer vision which is "understand the physical world through [images]."

If training CNNs does it faster/better than having a better understanding of Stereopsis/etc... then who cares?

It is easy to introduce different measure which will make deep learning fail and which will improve the state of the art and humanness significantly. They are: 1) Learning from small amount of examples, make training database very small with 3-4 pictures by class. 2) Learning across classes. Train on evening pictures and learn to recognize daylight. Train on direct sunlight and learn to recognize on side light. 3) Learning of entities. Train on entity in white and learn to recognize a combined picture. Deep learning community intentionally do not consider these tasks, they prefer to increase the training data size to the limit only huge corporation could afford doing a research.

Deep learning is currently the best way to solve tasks such as your #1 example - there are great results with learning to identify classes from even a single labeled example; and the proper (only?) way to do this is to use basic feature-extraction layers made by showing the system a bunch of unlabelled images.

> (1) a different measure for results, or (2) give an objective mechanism for evaluating the "worth" of a technique that doesn't involve looking at results.

One does science also to understand why something works (i.e. what is the theory behind it). Deep learning gives something that (perhaps) works, but doesn't give an any clue to the question why it works - on what theory does the neural net internally compute in its black box.

Based on this criterion one can even argue that deep learning papers should not be considered as science as long as the authors of the deep learning paper make no really hard attempt to present at least a partly understanding of the internal theory that the trained neuronal network executes (i.e. a very detailed interpretation of the weights with ideally falsifiable predictions). This is hard work, I know, but isn't that nearly anything in science?

Completely agree with this viewpoint. Explanatory power and reproducibility are two of the most important aspects of a scientific paper. Most deep learning based papers lack both these properties.

> Explanatory power and reproducibility are two of the most important aspects of a scientific paper.

The reproducability problem should be solvable if the authors would simply release the training data and source code that trains the network (as yould scientific practise would require). Or is there anything else that is necessary to assure reproducability (I'm not that deep into CV papers)?

Benchmarks are indeed valuable, but I think one of the points in the article was that if the field focuses too much on them, research with theoretical depth gets excluded. However, those theoretical papers have value that is hard to quantify--it may take years to see it!