Level 4 for example could be a blood bath, where cars drive all the easy situations, and hand over emergency control to out of practice drivers barely paying attention at a moments notice.
I believe our driving skills were better when we had manual transmission, and were more involved in overall operation. Hard to determine if safer, b/c of advances in crumple zones, airbags, etc as manual declined.
Level 4 requires that the vehicle safely stop in situations it can't handle.
The extremely sad part is that we could reduce the number of deaths until true AI arrives using simple tech that could detect if the driver is drunk or tired, and controversially forcing the speed limits(the car knows that this road is max 90Km/h and that the driver has less then 1 year experience so it will not let you get over 70 (local laws apply) unless you activate emergency mode - this would trigger some bad consequences if it was no emergency)
Driving assistants would improve safety certainly, but removing the driver and letting the assistant drive will need real world like tests, for now maybe you could have an independent group monitoring a number of Teslas and checking each time a human intervened and give a score on how bad the AI was.