Yeah it seems like you'd want to interpolate along an equiluminant plane using a colourspace that separates chrominance and luminance (and is ideally perceptually uniform)