This is also why it's always very, very wrong to compare potential faults of automated cars to humans as in "the automated car is X percent safer!", becuase it ignores the fact that mistakes in automated systems, at least as they are built now, are highly correlated.
If there is one bug in an ML system that is rolled out to an entire fleet that results in an unknown weather condition leading to fatal crashes you may create mass carnage.
Human driver errors are not correlated like this, which makes them much more robust as an ecosystem.
According to NHSTA, 144 drunk-driving deaths on July 4, compared to 36 on average: https://www.bactrack.com/blogs/expert-center/35042821-the-mo...
I would say that's mass carnage.
Human driver errors not correlated my arse!
The threshold issue is remote updates of car software. And Tesla had made that more mainstream and attractive to other manufacturers.
State-sponsored hackers are not going to ignore the opportunity. They probably have the capability already, in dozens of countries, and are just waiting for orders from the leaders. If war is starting, the order will be given. Sanctions could be enough to trigger it.
I could see arguing with the inevitability of the exploit -- Win95 botnets seemed much more inevitable to me than this Tesla mothership threat does. But it seems like you're arguing that they will both have similar impact if exploited. That doesn't make sense to me, because they're completely different threats, but it's possible I'm misunderstanding your argument in some way.
Imagine even 0.1% cars being controlled, the mayhem and loss of life that they can cause is just immense.
Power plants are also dangerous targets in a similar sense, but hopefully there is network separation for control services.
[1] https://www.wired.com/2016/08/jeep-hackers-return-high-speed...
Been a while since I looked in the details but from what I recall only very limited, well scrutinized communication is allowed to bridge the Ethernet subsystem over to the CAN bus.
I guess those risks are a bit like nuclear launch risks. Someone, somewhere (that controls the right key) could unleash an enormous accident. You just hope the system as a whole has enough redundancy for it to be sufficiently unlikely. In nuclear arms technology we have multiple authorizations required for launches. Then of course the underlying system has to be robust enough so none of those measures can be bypassed.
There is no fundamental barrier that I can see -- we know how to build incredibly robust systems using methods like formal verification of software and just thorough quality assurance and testing. In a way, much of the world is already exposed to those kinds of risks: most individuals's phones, laptops and even industrial computers can be remotely updated and incapacitated, exposing a risk of generalized trouble.
The adequate amount of resources to be allocated on those risky but extremely remote scenarios is what's important. I think there needs to be oversight guaranteeing every system is getting verification, testing and attention proportional to risk for society. The usual punitive incentives don't work very well for those cases. I don't think we have any agencies with this wide of an outlook currently.
And probably other manufacturers will use their own designs.
Unfortunately, this is one of those issues where we have to be lucky every time, and the bad guys only have to be lucky once. Given the number of different manufacturers whose systems have demonstrably been compromised in the past, the odds of avoiding catastrophic compromises in the future as cars gain more autonomous features don't look great here.
One plausible alternative is that we don't deploy systems like this, which have the ability to cause widespread damage including loss of life in moments, until we have worked out how to secure them properly against a single point of catastrophic failure like that. There are things that could be done to mitigate that threat in the meantime.
This is set against the knowledge that existing human-driven vehicles are involved in many tragic accidents per year, also causing widespread damage including loss of life. But there are other things that can be done to limit the damage there as well, without relying on autonomous vehicles as a silver bullet.
Most vehicles won’t let you reprogram the firmware without power cycling the car. Disable sensing of a crash is definitely its own ECM that is on a high priority bus. I am assuming your common <$40k car. When you head into bmw, merc Benz land this statement changes slightly.
> Villainess: I want every with chip with a 0-day exploit in a two mile radius around that motorcade now.
> Computer guy: There's over a thousand of 'em
> Villainess: Hack ‘em all. It’s zombie time.
> [zombie cars drive around causing mayhem]
Both the Prius[1] and the Model 3[2] had similar software bugs related to their anti-lock brakes. Both companies had a software fix a few days after the bugs were discovered. Tesla's fix was pushed out immediately to every vehicle. Toyota couldn't push out a fix. They had to issue a recall and have a technician update the software whenever that car ended up being serviced. How many months or more likely years did it take for every Prius to be updated with the software? How many miles were driven in cars that were known to have faulty brake software because it was hard to update them? You have to consider situations like this when discuss banning remote updating.
[1] - https://www.networkworld.com/article/2245704/toyota-to-recal...
[2] - https://www.wired.com/story/tesla-model3-braking-software-up...
Isn't that exactly what we're doing? If someone pushes out a malicious change, do you know how long it would take for that change to propagate to the entire Toyota fleet?
It wouldn't.
This gives all the more incentive to get the software correct in the first place. The model of "get the software as bug-free as possible upfront using stringent processes, testing, formal methods, and not using software in the first place when it's not actually needed" is better than the model of "put software into as many components as possible to make it shiny and get the software good enough to release before our competitors and play whack-a-mole on the bugs later through updates". Instantaneous updates make it easier for an attacker to take control of the update infrastructure and push an update that will trigger a mass-crash of cars during rush hour. When people have to asynchronously take the car to dealerships over many months, it makes this attack harder to go undetected.
I think you've just identified the root cause of at least one set of problems. The essential control systems in a vehicle should ideally be separated from other vehicle functions to prevent interference, whether accidental or deliberate, from compromising vehicle safety. The approach now being taken by manufacturers with their always-online, increasingly automated cars may be undermining that separation, without necessarily having adequate safeguards in place to ensure safety and reliability are maintained.
Anyway, this is moot in most cars, as they don't have sold driving software anyway.
In this thread, I'm simply observing that there can't be a clear separation between control systems and information systems if we're going to achieve these kinds of autonomous functionality. We need a more nuanced solution.
- know of the existence of Bosch,
- be completely ignorant of the company being an absolute giant in the engineering space, and
- be confident enough that they're some tiny power drill manufacturer to mock them publicly without pausing to look them up
It's like hearing someone say "the guys who make the Xbox are providing cloud services for the Pentagon? Who is running that ship lol".
-
Meanwhile there's the company who took Mobileye's LKA, which was already in cars with the same hardware but not continuously enabled because of inherent flaws related to non-moving objects... and turned it on all the time so they could call it Autopilot.
Then it killed some people because of the flaws that kept other manufacturers from using it the way they were.
And they're writing the software for self-driving cars? Who's running that ship lol
-
In a better world people with experience with safety critical stuff and the culture for it would partner with companies like Tesla to create something like a "plug in" system for SDCs. Where the safety guys could focus on defining a minimal viable envelope of operation the same way existing ABS and ESC systems mesh with drivers. Something like if the car is less than 100ms from crashing intervene separately of "normal" object avoidance
(And before someone nitpicks that's an arbitrary number of ms, yes there's no "isCrashing" variable, and yes it would be hard work to define how SDCs would handle intervention, but people crashing into parked firetrucks is worthwhile "hard thing" to solve)
https://en.m.wikipedia.org/wiki/Ball_Aerospace_%26_Technolog...
The brand that could cause the most damage is probably Bosch, a major automotive component manufacturer.