But critically, it's (probably) impossible to get past the "good as humans" point without a huge amount of learning data. That means either a lot of time or a lot of testers.

It's interesting that the new Tesla software will be shadowing while the meatbag is driving: not just reporting back driving data, but also simulating what it would do all the time. There's going to be a lot of situations where it would be wrong, all nicely laid out for the developers, ready to fix.