Two things: 1) humans do it by vision alone, hence, it is possible in principle, and 2) lidars are expensive and introduce one more failure point.

Most of the self-driving cars you see in bay area with lidars on top actually use those lidars to produce labeled data for vision-based distance estimator networks