You could adjust the error function. The common Root Mean Square error pushes predictions to the average. If you use absolute errors, or even a logistic function instead, you'll encourage the model to commit to a decision on a multimodal distribution.

Alternatively, use a discrete colour space and consider colours as categorical data not implying any ordinal scale.