There is already the color channel dimension in addition to the spatial dimensions and time dimension. Adding depth doesn't make things much harder, especially because there is so much correlation between depth and color and time.