I don't think it's attacks we need to worry about (there's even an XKCD about dropping rocks off of overpasses). The issue is that without good depth and velocity data (so probably LiDAR) there are lots of fairly common situations that an ML algorithm is likely to have trouble making sense of.