It seems like it should be similar in capability to a wavelet-based approach to e.g. image inpainting. In other words the neural network architecture essentially establishes a basis of features onto which the image is projected. Gabor wavelets are known to occur in the human visual system as the 'image elements' in a similar setup -- Gabor wavelets are kind of optimal, but clearly the features that this nn architecture uses are pretty effective too.