RGB<->YCbCr is a linear operation, so a deep enough network shouldn't have trouble learning to do the transform if it's useful. Gamma correction might be important, though.
Sure, but surely you'd rather not? I imagine you want the network encoding the details of the specific image, not wasting weights to learn about easy wins that apply to every image. You could burn a whole layer converting to YCbCr, and the net in the article only had 3.
You don't know if that is actually the case, the whole appeal, originally, is that you don't need this kind of handcrafting of features. Now, clearly, people preprocess their data all the time so that point is kind of moot.
However, if YCbCr was meaningfully better for the network, I think it would be pretty well-known as a preprocessing step. Also, the actual sensor data is RGB.
Re:Exploring Neural Graphics Primitives
by limbicsystem on Thursday March 30, 2023 @05:28PM
(#35379038)
But the human visual system is insensitive to high resolution yb and red/green information so you can essentially down sample those layers to almost nothing before you even start. That's a fundamental trick which is also used in another way by jpeg but presumably not by this RGB algorithm.
If you extend the JPEG analogy all the way and consider the 4:2:0 subsampling mode, then you could reduce your input parameter counts by exactly 50% by consuming a pre-converted image.
Turning RGB into YCbCr is a pretty essential step for achieving efficiency gain in techniques like JPEG. Quantization works a lot better when we can do the luminance and chrominance separately.
If you want the AI to "see" more like a human, converting into this kind of color space is an important first step IMO.