It seems like it should be similar in capability to a wavelet-based approach to e.g. image inpainting. In other words the neural network architecture essentially establishes a basis of features onto which the image is projected. Gabor wavelets are known to occur in the human visual system as the 'image elements' in a similar setup -- Gabor wavelets are kind of optimal, but clearly the features that this nn architecture uses are pretty effective too.

Somewhat similar to content aware fill in Photoshop [0]. The untrained network can latch onto frequent patterns and match them to holes in the data.

Why doesn’t it paint everything white? Are these actually transparent images or are they somehow tagged?

[0]https://helpx.adobe.com/photoshop/using/content-aware-patch-...

Yes, for the inpainting, the parts to be painted (big white deleted areas) are supplied as masks, so it doesn't try to match them.

But its generating unique content in those areas...

Yes, every "run" of the network is generating pixels in those areas, but they're not being compared against the white (deleted) pixels. On the final run of the network, they're still not being compared against anything, except by us, visually, when we look at those pixels.

Probably this is a consequence of least-squares training; the 'average' solution is extremely unlikely.