Humans can reason about things that haven't to them happened before. Today's machine learning systems cannot. As you say, to react appropriately they must have been trained to do so using human annotated data.
The argument against FSD is that you would need an infinite number of annotated examples, and an infinite number of subroutines for behaving in any identified situations, because the space of driving is effectively infinite.
Until machines are able to do general reasoning about things they've not experienced before then FSD is not happening. By the way, Demis Hassabis thinks that this sort of transfer learning is the key to solving AI.
That’s not how machine learning works. You can “train on exceptions” as much as you want, and you have 0 guaranteed results. It can help, it can make no change or it can cause unexpected regressions.
Media (and surprising amount of tech people as well) tend to claim that ML learning is like human learning - repeat something enough times and you’re done, you know how to do it. ML is no where close to that point.
Prove they are getting better. They run into the sides of trucks and off ramps on the freeway quite often (and don’t you dare blame the driver... it’s “full self driving” remember?)
Can you prove a machine learning algorithm does the right thing in novel cases it hasn’t encountered before? Nope.
And also, don’t reply with “well can humans”. That is a lame rebuttal. Computers will be held to an almost 100% non-failure standard before society accepts them. And that will never happen because of, well, reality.
Most people wouldn't instantly know how to handle those cases either. Many people would obey the drunk guy. Maybe that's the right thing to do.
None of those are edge cases because every time it drives it will encounter some novel edge case that has never happened before and it will have to perform better than a human.
That's why these things could never be rule based, there are too many small exceptions. When they're not overfitting/able to memorize all your training examples, neural nets learn heuristics, just as people do. Different people learn different heuristics. Granted, they don't have much of the same context about the world that people do, it will take a long time to build enough examples for them to infer all of that. But Tesla's fleet is getting more driving experience every day than you will get in your entire life, and every time they train on one of those exceptions, the entire fleet will benefit.
Don’t even get started with liability. Once you take away the steering wheel the manufacturer is on the hook for every single mistake and every single accident. You’d be insane to be a manufacturer and sign up for that.
If drivers no longer carry their own insurance, this is probably going to be handled by insurance at the manufacturer level, and baked into the price. The insurance will demand certain processes to prevent large-scale bugs being rolled out.
I don't see any fantasy here, just a lot of work.