Not an expert so take this with a grain of salt; I could be misinterpreting the paper.

It seems that the current accepted method is to train a network with distorted images as the input and the correct undistorted images as the targets. Then after training you can feed a new distorted image into the trained network and get the estimated "fixed" image.

However this team actually uses the distorted image as both the input and the target to the net. So if they were to let the training go on for too long the network will produce an exact copy of the distorted input image. But for some reason, the structure of the network means that the estimated output learns realistic features first, and then overfits to the noise afterwords. So if you stop the training early, you get an image that incorporates realistic features from the distorted image, but hasn't had time to "learn" the noisy features.

This is fascinating because I've been running into something similar with sequence to sequence models translating natural language into Python code. I got better results stopping "early" when the perplexity was still quite high, I thought it was a little crazy.

Fascinating! Could you share more on the details of your experiment?

> So if they were to let the training go on for too long the network will produce an exact copy of the distorted input image.

It won't. The objective (min (E(x, x0) + R(x)) they are trying to optimize (over output images x) amounts to a combination of:

- The output image x should "look like" the input image x0, this is the error term E(x, x0);

- The output image x should be "regular", this is the regularization term R(x).

The latter term prevents overfitting: R should be chosen such that noisy images for example are considered irregular (high R(x)).

Ah you're totally right about it not producing an exact copy, they even mention how they used different error functions for the different classes of distortion, my mistake.

But w.r.t. the regularization term, what I thought they meant by "we replace the regularizer R(x) with the implicit prior captured by the neural network" at the end of pg 2 was that they let the natural behavior of a neural network during optimization serve as the regularization, without need for an explicit regularization term. Not entirely sure though.

> for some reason, the structure of the network means that the estimated output learns realistic features first, and then overfits to the noise afterwords.

Why does this happen? What characteristics does the network structure have that cause this effect?

I think the reason this paper is so interesting is that no one had any idea why this happens

Wow thanks for the explanation, I would they would just use what you wrote for the abstract.

At first, I thought the machine learning algorithm was able to just bridge the gap in images from nothing...

CSI time: Enhance ... enhance... enhance!

The structure of convolutional neural nets specifies much of the prior knowledge necessary for learning. In other words, the design of these neural nets makes a lot of correct assumptions about the nature of images (stationarity of pixel statistics, locality of pixel dependencies, and so on).

By structure, we are simply referring to the number of layers, the number of neurons in each layer, and the specific connections between neurons in each pair of neighboring layers, right?

So in this paper, they carefully chose a certain structure, set the weights randomly, and then what happened after that? I understand that they did not then train it with a training data set, but I'm not quite getting what they did with the single distorted image.

Well given that it's CNNs, you're leaving out weight sharing.

So by structure you should also include the demand that the prediction of any NxN patch of the image should be roughly equal to the prediction of any other NxN patch of the image.

is the structure of these CNNs learned or designed?

Do they run some kind of optimizer to learn the optimal CNN structure or does some person sit down and pick structures to include in it?

Convolutional layers are designed, by and large, and they're mostly the same everywhere. Yann Le Cun came up with them in the mid-90's, but their academic origins go back to at least the 50's and 60's.

then the results of this paper are sort of surprising that it's that good, if the structure is just relatively old, hand-crafted and traditional.

Handcrafted, but auto optimizing is a hot research topic right now with DeepMind et al. Needs to be evolved at deeper level than what they’re doing now so architecture is discovered, not just optimized.

By structure, do you mean the architecture of the NN?

Meaning number of hidden layers, nodes per layer, and their connectivity with each other and with the input and output layers?

No, the paper means primarily the weight sharing in each kernel filter within each convolutional layer, and the stacking of these layers in deep networks.

Which in turn relies on older work.

I wonder if anyone has looked at what Mumford-Shaw implications would look like projected onto a CNN?

Neural networks are powerful without even training them. By merely designing the structure of them you are creating something.

Truly, an occasion for this koan:

Sussman attains enlightenment

In the days when Sussman was a novice, Minsky once came to him as he sat hacking at the PDP-6.

“What are you doing?”, asked Minsky.

“I am training a randomly wired neural net to play Tic-Tac-Toe” Sussman replied.

“Why is the net wired randomly?”, asked Minsky.

“I do not want it to have any preconceptions of how to play”, Sussman said.

Minsky then shut his eyes.

“Why do you close your eyes?”, Sussman asked his teacher.

“So that the room will be empty.”

At that moment, Sussman was enlightened.

http://www.catb.org/jargon/html/koans.html

If there ever was one this is it.

Okay, I admit it, I'm not enlightened. Will somebody please Explain Like I'm 5?

Initializing a neural net with random weights =/= making a neural net with no preconceptions on how to play. It still has those preconceptions, you just don't know what they are.

Closing your eyes so you can't see the contents of a room =/= the room is empty. Not knowing what something is, is different from it not existing

Thanks!

This kind of reminds me of what I took away from Gerald Edelman's work ([0]).

The gist as I remember it was that he theorized that the cortex is composed of assemblies of neurons formed in development (in the womb, potentially with epi-genetic factors). Each assembly, just by chance of it's structure, is likely good at something. Over time, assemblies take up the tasks they happen to be good at and further specialize with experience.

[0] - https://en.wikipedia.org/wiki/Neural_Darwinism

Creating is perhaps better states as “asserting”

Could be wrong too, ML is not my field.

Basically, a generator neural network has two things which affect its output: an input and some parameters (weights).

Let's use the setting of the task for denoising images. They use a network who's output is to is the denoised image, and compare it to the noisy image to get a score for how good the denoised output was.

Now the strange thing is, for the input of the network, they just use random garbage. The only thing they move around to try get a good denoising score are the parameters of the network (not the input).

They find that by only adjusting the parameters, even with fixed random crap inputs, if they find the parameter setting which minimizes the noise, they actually still get great looking results.

This suggests that the networks they tested this method with (other researchers work who do well on this task) are based much more in the inherent structure of the network, rather than the model refinement from training on thousands of images, since even given random crap, they generates good results, as long as the parameters are tuned well to the noisy image.

Instead of looking for patterns from itself on others images (training data) it starts with noise and deforms such noise based on patterns found on itself and favors deformations closer to the input image; eventually reaching something close to the input image without the noise (cause the noise it's pattern-less or at least weak enough to die over stronger patterns)

Does this say more about CNNS fundamentally or our design process for their structure? What sort of assumptions might one make when "structuring" a CNN?