You could do the YUV to RGB conversion operation in your pixel shaders in the GPU. That way you save a bit of bandwidth compared to uncompressed RGB.

It's been done. There are even GPUs that support this operation natively, so there's no additional overhead.