I'd be very surprised if this is what it's to be used for. The technique requires color, depth and motion vectors. That's three separate video channels, and two of them contain data that isn't usually stuffed into videos.

Any compression artifacts are going to stick out like a sore thumb, so you'll need to stream very high quality, and you're going to have weird interactions between different layers being compressed differently.