I don't believe your hand coded features will outperform ML trained over a billion miles, even in the long tail rare events.
Sure ML is bad at rare events, but hand coded features also map poorly to such rare events, because it's unlikely you'll be able to imagine them all, or even manage to handle categories of rare events well.
Furthermore, I believe it would be possible to train an ML based rare-event detector, which simply detects the difference between "regular driving I'm familiar with" and "something funny is going on, I haven't seen this much in the training dataset", and then refer to a remote human to resolve the issue.
Successful remote control is a whole different project, very different from self-driving. In order to give the remote pilot enough situational awareness, you need plenty of streamed imagery (recreated or recorded). And you need a feedback loop with a very small latency. Not saying it couldn't be done, but it won't happen "organically" by cars becoming self-driving.
I don't think hand coded features are necessary. No one hand codes humans to drive.
We haven't heard much from Google because they've been programming and validating all the unglamorous little problems that training data and a neural net can't solve alone. 4 way stops, passing motorcyclists and stuff like that, where old fashioned hand coded features are necessary to adress situations in which specific kinds of reasoning is necessary.
Tesla has to solve all this boring stuff as well before they can offer comparable capabilities in an AV.