[0] https://www.zachaysan.com/cars and apologies for the length; it was a very stressful time in my life.
[1] http://support.blackberry.com/kb/articleDetail?articleNumber...
When you have 100,000 self-driving cars on the road, and someone figures out how to hack them, and drive them simultaneously into crowds, that will be a BIG BIG problem.
And this problem isn't just about self-driving cars. It is a fundamental issue with all algorithms. It's a lot harder to hack humans to do bad things (although it can be done with sustained messaging and propaganda). But algorithms can be compromised and exploited.
I'd say, it's probably easier to hack individual humans, but - like everything in computing - hacking algorithms scales well, while human factors generally don't.
First, they're not physically inspectable by every Tom, Dick, and Harry. This means it's easier to obscure interfaces and other potential areas of attack. Not state-actor proof, but probably ISIS-proof. Second, they only get software updates while parked at an airport, so it makes MITM attacks harder (though not impossible). Third, they're in the air where death isn't a sudden movement away. Pilots can override the system and fly it manually. Fourth, they're heavily monitored with errant flightpaths reported to militaries around the world (to stop another 9/11).
We have none of these safeguards for self-driving cars and there are going to be hundreds of millions of them.
https://www.computing.co.uk/ctg/news/3020901/dhs-team-manage...
How autonomous systems will make anything different? Why do you believe people aren't currently hacking cars?
I don't intend to be critical at all, I believe this is a very interesting point too that should be discussed. (I hope to check the links on my interval).
> How autonomous systems will make anything different? Why do you believe people aren't currently hacking cars?
It might make always on data connections near-mandatory (to get maps data, etc). At least in my car, the insecure embedded computers are air-gapped from the internet.
Part of the reason is that, in current ecosystem, companies will find ways in which self-driving requires being constantly on-line and connected to vendor's server. Off-line processing is not in fashion these days.
If we use a more narrow definition of weapon, e.g. a tool optimized for the primary purpose of injuring or killing people then it certainly is not a weapon.
It's not useless; the damage that can be caused varies by degree between things. Then you have to look at two cases - suitability for object to be used as a weapon, and the damage it can cause on accident. Unlike knives or hammers, both those factors are very high for cars.
The fact how dangerous cars is is very much underappreciated by people in general, as evidenced by the number of morons on the road. We already lose hundreds of people daily in the US alone because of this; now we're trying to add another class of drivers into the mix - algorithms written by greedy optimizers caring primarily for short-term profit and being first to market. This should give us some pause.
I'm not saying this technology is not possible or not wonderful, but I think the current ecology of self-driving efforts is unhealthy. We have a for-profit race by companies, many of which can't be trusted with getting software right, and most (all?) of them pursuing self-driving capabilities by means of half-understood brute-force black boxes the neural networks are.
> and the damage it can cause on accident
You are conflating (un)safety of a tool used in the way it is intended (kitchen knife = cutting a steak) with accidents (cutting a finger) and with malicious use (stabbing people). Those three categories are not the same for object-that-may-act-as-weapon and object-designed-as-weapon.
Conflating them collapses the number useful things we can communicate.
So are you concerned about Tesla intentionally building killing instruments? Or potential for accidents? Or the potential for intentional misuse?
> The fact how dangerous cars is is very much underappreciated by people in general, as evidenced by the number of morons on the road.
Cars also provide immense utility. If all they did were providing the thrill of speeding then they would probably be banned as too dangerous. One of the tradeoffs is the overhead of enabling people to drive. We could drive down the number of morons by requiring astronaut training for vehicle operators but again, that tradeoff seems too harsh and it's more efficient to occasionally let people die in traffic accidents than letting them die because nobody qualified as ambulance driver.
> We have a for-profit race by companies, many of which can't be trusted with getting software right
In the short term this may cause more deaths than necessary. But on the other hand it might be the quickest way to find a winner and then hold the rest to the same standard. As long as the experimental fleets are small they are just a blip in the statistics. Right now they should be equated to the yearly batch of first-year drivers who have an inherently higher risk profile due to lack of experience. We still accept them on our roads in the expectation that they improve.
What is important is to make sure that they are as good as or better than humans once they roll out in large fleets.
The latter two.
> We could drive down the number of morons by requiring astronaut training for vehicle operators but again, that tradeoff seems too harsh and it's more efficient to occasionally let people die in traffic accidents than letting them die because nobody qualified as ambulance driver.
I don't think this is the real reason. You don't need astronaut-level training for vehicle operators, just more than the ridiculously low standard of today, and more importantly, much stronger and harsher enforcement of traffic laws. I doubt that this will reduce the number of qualified ambulance drivers.
I suspect the real reason we tolerate so many morons on the road is path dependence. When cars first appeared, they were rare, slow and safe. In the couple of decades it took to get to the present density and speed of cars, it became a social status symbol, and something politically impossible to rein in.
> What is important is to make sure that they are as good as or better than humans once they roll out in large fleets.
I'm afraid that with self-driving tech based on neural networks, with no ability to inspect and verify what's going on, we'll eventually have to eat the risk and roll them out in large numbers before we know they're as good as humans.
I do not agree that neural networks are a "black box" with "no ability to inspect and verify". Even putting aside the many methods to understand what a neural network is doing without running it, at core, neural networks are well tested instruments. That's how they learn-- by testing themselves.
Obviously it's possible for a neural network to have odd behavior in circumstances not accounted for but that was always going to be possible at the level of complexity we're talking about here.
We're talking about cutting edge technology here -- and I agree with your general sentiment. I just don't agree with pinning the blame on "... based on neural networks". The same factors would apply to any codebase of this complexity.
Name three :).
> neural networks are well tested instruments. That's how they learn-- by testing themselves.
Last I checked, neural networks are well-tested in a sense that if you throw a big database and a shit ton of compute at them, they'll learn to accurately work within that database. Step out of it, and all bets are off. We're better at this than we were 30 years ago - good enough to apply this technology to consumer-level products in which mistakes don't really matter. I'd be wary of applying even current neural networks to safety-critical tasks.
> Obviously it's possible for a neural network to have odd behavior in circumstances not accounted for but that was always going to be possible at the level of complexity we're talking about here.
The problem is that with NNs, the odd behavior is usually totally unexpected, and you can't really inspect the network beforehand to discover the possible ranges of error-generating inputs. Everything works fine but every now and then you get a patterned sofa classified as a zebra, or a car + little noise classified as a toaster. And then there's no obvious relation between multiple misclassifications, because the reasoning structure of the neural network is implicitly encoded in its weights.
> The same factors would apply to any codebase of this complexity.
I think there's a fundamental qualitative difference here. A codebase can be complex, but ultimately it has a structure, and usually (in case of ML) represents a well-understood mathematical structure. Neural networks have simple code, and the whole complexity is hidden in opaque matrices of numbers, where even single changes usually have global effects.
I'm not trying to dismiss NNs in general; I just don't trust them in applications where health and safety is at stake.
I'm sure that making the primary means of long (greater than walking) distance transportation for the majority of the population more expensive and higher stakes is going to work out great in the long term. I can see the parallels with healthcare. Creating yet another part of life where a single screw up that is capable of ruining the financial well being of someone living slightly better than paycheck to paycheck is not going to do positive things in the long run.
But who determines which is which? The letter of the law really, really, really sucks when it comes to traffic law. I haven't collected data but I'd wager that pretty much nobody follows the letter of the law for an entire drive from A to B
>For example, treating speed limits as suggestions instead of hard constraints
That's more human than reckless. Outside of places with Orwellian enforcement (I'm looking at you Europe with all your cameras) they really are. The vast majority of people go at a speed they feel comfortable in the conditions. This is why (conditions permitting) traffic flows at 80 even when the sign might say 55. People only follow the speed limit when they feel it's a comfortable speed. This is why the general recommendation is to set speed limits for the 90th percentile speed. If you don't do this on highways you get people doing the (inappropriately low) speed limit in the wrong lane. Passing on the right, tailgating and all the other things caused by traffic friction which is more stuff for drivers to keep tabs on and that decreases safety for all. There have been studies on this (Google "traffic friction" and filter out everything that has to do with literal friction). If anything speed limits on multi-lane roads should be raised to reflect the speeds people actually drive. I hope we'll see more dynamic speed limits in the future since they'll help a lot.
>overtaking in places where it's not allowed.
If every 100th instance of an illegal pass (usually on the shoulder when waiting for someone who's stopped to take a left turn or on the right on the highway) in my state resulted in a ticket it would probably be about a year before half the state's drivers hit the three strikes cutoff and had their licensees revoked.
The picture I'm trying to paint here is that aggressive enforcement of existing laws would probably be bad for the population at large because people break traffic laws in inconsequential ways all the time and that stronger enforcement of them would just screw people over (unless of course you enforce them so well that important people get screwed which would result in the laws changing) and discretion isn't an answer because that just results in profiling.
America banned alcohol by an amendment, pretty much the hardest political barrier we have
It's just an excuse IMO
Any object that can be remotely programmed to drive itself at high velocity into a target, yes.
A person can explain "why" they did something when an accident happens. I've spent a lot of consulting hours helping unwind the "why" of ML/AI models, and we bill regardless of whether we can actually find that answer.
Putting aside by personal bias that I think Tesla is functionally incapable of doing anything well, lets pretend they deploy FSD - I think the fact that in that 1 accident to every 100+ human accidents you can't find "why" will override the fact that there are xx% less accidents.
Just from my experience consulting in healthcare and working with actuaries during the Obamacare debates and modeling out the impact of that -- people just honestly aren't open to statistical arguments when it comes to human life or emotional issues.
(Not saying they're wrong, me and my wife completely disagree about so many issues where it's emotions vs statistics, I think she has a very valid point on many of those - what I'm trying to say is this is going to be a philosophical debate more than a statistics debate so Tesla should gear up accordingly).
Also, while I agree that you can "program" a human, you can't program them to jump up and down and touch their toes (so to speak)
the truth is, people have a lot of experience with other people. we can usually predict behavior of other drivers on the road, and can even predict pathological behavior or know how to handle it. that is a major point. when machines and software fail, they can fail in big, unpredictable ways. humans fail in fairly predictable ways. none of us have this experience with neural networks or machines.
i barely trust my computer to work on a daily basis, much less a vehicle.
and lastly, when a human is driving, fault is usually clearly and easily assignable. how fault is assigned in the case of machines driving is not clear at all.
of course safety is important. and i agree that verifying driving ability is well behind where it should be. elderly and inexperienced and inattentive and just plain bad drivers are major problems on the roads.
and fault certainly does matter. aside from legal responsibility, we are emotional beings. there is a difference between getting hit by a drunk driver versus a true accident. that affects people in real ways. as a motorcycle driver, i will be terrified of these automated vehicles. i am already terrified of human drivers, but i can often predict their idiocracy.
no one wants to be killed by a machine, especially one built by wide-eyed engineers.