>But I still feel that it would not solve the averaging problem. It would hop between different colors and probably converge to brown.

At least to the extent that GANs work, it works. They will alternate between the observed colours based on the noise vector. They do not simply converge to averages, because the discriminator easily recognizes brown apples as fakes.

It could try to classify the apple tree or the context, but it would require a lot of training data. If it's out of context, it should select a color based on probability. But it's hard to solve this with just input and output data. The simple solution is to use noncontradictory training data, i.e. only having green apples.

I have an urge to teach it simple logic. Instead of making it brown, it selects the color with the highest probability from a range of colors. However, I haven't come across a deep learning implementation like this to mimic.

Although it is quite egregious here - this is not a problem inherent to colorization but rather to generative models in general.

Using something akin to a variational autoencoder would solve this problem, because it learns a distributional approximation rather than a single point estimate of the color, and then the random noise vector input allows one to sample from this output distribution. Similarly, Mixture Density Networks allow you to model a distribution and then sample from it.

You could adjust the error function. The common Root Mean Square error pushes predictions to the average. If you use absolute errors, or even a logistic function instead, you'll encourage the model to commit to a decision on a multimodal distribution.

Alternatively, use a discrete colour space and consider colours as categorical data not implying any ordinal scale.

> If it learns that an apple can be red, green and even yellow - how does it know how to color it?

I dunno, does it look more like a red apple, a green apple or a yellow apple?

Google "learning xor".