Good point, and it isn't just vehicles with auto-pilot perhaps all vehicles.
The 2017 and 2018 Chrysler Pacifica Minivan kept on randomly shutting off[0] and loosing all electrical power at freeway speeds. You had to put the vehicle back into park, on the freeway, to restart it. There are several stories of near death experiences trying to get to the shoulder. It was intermittent.
This is relevant because nobody could figure out why, even the manufacturer. They had to install custom self-powered diagnostic equipment in consumer's vehicles to even try and get enough data on it. Turns out that suddenly cutting power to the onboard computers doesn't make them very useful for diagnostic analysis, some dealers even acted like the problem didn't exist because the computer had no record.
They did ultimately get it fixed (replaced an engine management unit or similar) but it does go to show that modern vehicles are getting so complicated that even manufacturers are struggling to stay ahead of it. Tesla's autopilot and similar just takes that up another notch.
[0] https://www.autoblog.com/2017/11/21/chrysler-pacifica-owners...
A lot of that is for the infotainment systems, but regardless, we're probably going to be seeing automakers struggle more and more with all of the problems that have plagued the software industry for its entire history.
On the one hand, I want to say that situations like this Tesla crash and your minivan example get so much attention because that's rare, so the manufacturers are mostly getting it right and we'll all be fine. On the other, knowing what I know of the software industry plus my past experience as a shadetree, the prospect of that much code running in 65-mile-an-hour murder machines gives me the shivers.
It's sort of natural - it's "why not". We can afford to have more SLOC than ever before. Why bother to try to reduce it, that's not a priority, our priorities are features and fixes. It becomes too much to really understand and manage, and then it's all automated checks and automated tests.
It's very hard to get the average/common developer to spend time thinking about how we can remove stuff, clean stuff up. And managers don't want to pay them to do that. In some places it eventually becomes a big problem, and then it's time for a big rewrite. Most software is throw-away. Website front-ends, mobile device drivers, etc ... mostly thrown-away roughly every year. It's the relatively rare and stodgy classic open-source stuff that's re-used (with minor updates) for decades. OK, I'm getting off-topic ...
I'm not sure what "verifiable models" means here. There are a bunch of people using Simulink to "auto-generate" code for some ECUs. I've used it with great results, and I've seen others do that as well. And yet some people still create horrible things with it just like regular code. But not once did I see anything that sounded like formal verification. That doesn't mean it never happens but the way The Mathworks markets their tools, you'd think some kind of magic is done automatically.
On a related note, I always laugh when Mathworks promotes the ability to compare "simulation" results to that of the "generated code". They call it "software in the loop" testing. This is actually getting their customers to verify that the tools they provide work as advertised.
For that kind of development I'd actually prefer to just shut the simulator off always generate code. Simulation should only ever be for plant models, not control systems you want to actually implement in software.
To me formal verification has always meant proving correctness in a mathematical sense against some sort of formal specification.
None of that really matters though because AFAIK you can't prove a neural net has been trained "correctly" and doesn't contain errors.
Progress is being made in this area. Here is an "old" paper: https://arxiv.org/abs/1702.01135
This is more a hijack than a direct critique but I think the general issue is the assumption that we need a private vehicle for that.
Trains are perfectly capable for bringing me from A to B while I am sleeping for years now.
Edit:// with years I mean years. Before that pressure made longer rides usually more uncomfortable than they are now with modern trains. Probably only because my area has plenty of mountain tho.
In most parts of the country we don't have dedicated passenger tracks. Amtrak (our passenger line) just rents track time on the freight lines. The problem is that freight lines are often single track lines so if there is oncoming freight you just wait until the freight clears. Sometimes this his hours.
Until we have dedicated passenger lines, long distance train travel is a non-starter here. It was too slow for my 75 year old uncle on a site seeing tour. His exact words were "fun, but never again"
I think demand for trains has diminished which is why they're not as nice as they could be; However I found plenty of nice trains on offer in Europe and Asia.
For a very limited subset of points A and B, though...
Getting to, into and away from the airport easily ads another 2 hours tho. Also this route has 3 flights a week compared to a train every 2 hours.
With the train I just get in and enjoy nice service, free power and WiFi and usually get some work done.
Not to mention the difference in my ecological footprint.
My point is that it's not like you are wasting time in a uncomfortable environment usually. So I dont mind taking the train, actually usually even prefer it, so do millions of people all over the world.
Only as clean as the electricity is produced. In many countries electricity production relies on coal, petrol or gas, which are far from being clean energies.
Tesla actually has that, it's not widely documented but you can do a voice command of "Report bug" followed by a brief audio description and it's uploaded to Tesla.
No idea how it gets triaged but I wish more products had a similar feature.
Seems an obvious way to improve: look at where your drivers are overriding your system.
When I say that it's at least ten years away and likely much longer because it will take big breakthroughs (strong AI, lot's of knowledge about the world, etc.) they look at me like I'm crazy.
I really hope it happens in my lifetime, but I remember talking about Greenblatt's chess program (see [1]) when I was at MIT in the 60's. The AI folks really thought that a chess program would be world champion within 10 years. They finally did it, but it took more like 30 years, and one of the reasons they were successful was that Moore's law eventually allowed a super computer (Deep Blue, at the time the 259th most powerful computer in the world) to evaluate 200 Million board positions per second[2]. Deep Blue's opponent, world champion Garry Kasparov) was most definitely not evaluating 200 Million board positions per second.
I see driving as a much more complex task than playing chess, so maybe 30 years before we will put our children in cars to be driven across town. I will be happily surprised if it arrives sooner.
[1] https://dl.acm.org/citation.cfm?id=1465715
[2] https://en.wikipedia.org/wiki/Deep_Blue_(chess_computer)
It may take 30 years to handle blizzards and every edge case but you can just park the car to avoid those problems temporarily while still serving 99% of the market so they don't need to be solved now.
Update: Since Oct'16 self-driving software is an add-on option and Musk has been tweeting[1] since Jan'17 that it's just months away and that its autopilot is somehow safer than adaptive cruise control. This Oct'16 video even starts with the claim that the driver is only there for legal reasons.
There is a lawsuit since Apr'17 regarding above - Google dean sheikh et al v. tesla, inc
[1] https://twitter.com/elonmusk/status/823632597284691969 " Yes, safety should improve significantly due to autonomy features, even if regs disallow no driver present 23 Jan 2017"
3 months maybe, 6 months definitely 23 Jan 2017
They are not careful at all. The first text on that page says "Full Self-Driving Hardware on All Cars"
You have to be moderately savvy to realize that you'd also need this to be coupled with "self-driving software", which the Tesla does not have.
By doing this they are in my opinion sabotaging the invoations they are actually making.
My only experience of automated driving is the feature where you can set a maximum speed. This is useful in the UK since average speed cameras are commonplace on the motorway network nowadays and it is easy for your concentration to lapse and drift over the current limit. The cameras are automated and there isn't much leeway.
However, the speed limits often change in range of 50, 60, 70 mph depending on local conditions and roadworks, so you can never just 'set and forget' you need to be alert all of the time since speed limits can change frequently.
It's going to be a long time before I trust a car to do this on its own without my supervision and, as regards going to sleep and letting the car do 'everything' on its own, I don't envisage being able to do this comfortably in my lifetime.
If it's any consolation that's not how autopilot on an airplane works. Pilots are expected to remain alert and ready to intervene in case of trouble.
Ideally, this would be sent to the NTSB as well to determine if they need to issue a recall and possibly also shared with the public if the car's owner is okay with that.
Issuing a bug report to the manufacturer is fine if the manufacturer is being responsive, but fixes take time and large companies don't always put a high priority on fixing bugs. Sharing bug information more widely increases their accountability and I think potential buyers have a right to know about safety-critical flaws before they buy a car.
For babies car is autonomous already. They have to be strapped pretty tight in a bit reclined position.
Do you mean to sleep sitting just like passengers in a car kinda can? I guess people are able to sleep that way. I for one have to be totally exhausted to fall a sleep sitting.
One certainly can't hope to sleep laying on a side under a blanket. Self driving can't ensure that car will never experience sudden deceleration. For the same reason you will not see seats facing back of the car. Because you could be killed by your or other passenger's laptop or burned by your hot coffee.
I wonder if we would have had safe planes if that was the mentality when they were first invented.
Something that will also be great will be a sort of "crash dump/bug report" button for these cars. If at any time your car does something unsafe, you can hit that button. The car will save the last 60 seconds so the manufacturer can analyze it to debug and figure out what went wrong.
I was so excited about auto-pilot and dreamed of getting in my car and sleeping while it made cross-country trips. So much for that, that seems way far out.