the images need texture and edge detail to pick up on depth. Any portion of an image that is a solid colour would need to be interpolated in some way, possibly inaccurately. Occlusion is also a bit of a problem resulting in gaps in the depth map that need to be filled somehow.

here's an old comparison of algorithms. I imagine the state of the art has improved with Deep Neural Nets recently.

http://vision.middlebury.edu/stereo/eval3/

edit: surprise! the page appears to be kept up to-date with new algorithms and recent techniques, and indeed the top performer is from 2018.