> My guess is that using color sensors would have either increased the 3D mapping precision or added weight/power/bandwidth requirements, or otherwise been less robust in that environment.

I think you meant to say decreased? In which case I think you would be correct! Camera pixels are made up of these things called photosites which don't by themselves record color, only brightness. In order to record color information, the photosites are placed behind a Bayer filter[1], which effectively reduces the resolution of the camera by 3, because in order to get the color of a pixel you need its red, green and blue component. Bayer filters also frequently have a small blurring filter in front of them to make sure that nearby photosites with different color filters get the information they need.

If you're looking for the highest resolution image possible, black and white is the way to go!

[1]: https://en.wikipedia.org/wiki/Bayer_filter