If you’re talking anything below level 5 then all we will be doing is making current human drivers worse. Some of the deaths so far have been people who knew the limitations of the technology and still got complacent. If it requires you to have constant supervision then people simply won’t do that and will die.
The real limiting factors for level 5 are less legible to humans. We understand the physical world via years of experience and good priors. Teslas crash straight into concrete lane separators and Ubers hit pedestrians that they know are there. In the latter case the actual reasons for the collision would strike a human as absolutely nuts - things like the algorithm not having a proper sense of object persistence or being terrible at commonsense reasoning. Yes, the Uber system was quite primitive, but the way that it broke gives us some insight into how AI drivers face a totally different set of challenges than what we would think of as "difficult conditions".
(Disclosure: I work for Google which is a sibling of Waymo)
The radar can penetrate fog, rain, etc. https://navtechradar.com/features/autonomous-vehicles/
Ultrasound works well up close for detecting a potential collision.
In theory you could have all four.
20 years is a long time, arguably you're already wrong because self-driving vehicles already exist. But Self-driving with the ability to replace Uber drivers will be here by 2030 at the latest.