Chrysler has had similar issue[0] this last year, their Pacifica Minivan had a software crash that would cause the engine management computer to crash and the vehicle to completely lose power while operating (inc. at highway speed). In effect you'd have complete loss of electrical power and even power steering, brake assist, and hazard warning lights(!) would be lost.

Took them six or more months to fix it as they weren't able to reproduce the software glitch.

[0] https://www.nytimes.com/2018/01/12/business/chrysler-pacific...

Why is the infotainment system even on the same computer as the brakes!?

Modern electronic cars use a single bus (CAN bus) that connect all electrical systems. The brakes, the engine, the wipers, the stereo...

Luckily it is a very well-tested system, it was one the first commercial use cases for QuickCheck [1], but still...

One of the bugs that John Hughes found was an integer overflow that transformed a lowest-priority message into a highest-priority message. [1] https://www.cs.tufts.edu/~nr/cs257/archive/john-hughes/quviq...

This blows my mind.

On the Stanford Solar Car Project -- a mostly-undergrad student group -- we always built our cars with two physically separate CAN buses, one for nonessential features and one for the motor controller.

That way, we limited our surface area for catastrophic bugs: the only things connected to the safety-critical CAN bus were the motor controller itself and the throttle board.

This for a one-off experimental vehicle.

CAN is a very simple, unauthenticated bus. Any device can send a message that every other device on the bus will receive. Messages are typically just a few bytes long, binary encoded.

The idea of attaching an internet-connected infotainment computer to the same CAN bus as the brakes is absurd. Doing so on a production car, even more so.

--

The Ford story is disappointing but not totally new.

A few years ago, Jeep committed similar design malpractice, resulting in a bug where Jeeps could be crashed remotely. https://www.wired.com/2015/07/hackers-remotely-kill-jeep-hig...

I'm surprised that regulators still allow these kinds of designs.

> CAN is a very simple, unauthenticated bus. Any device can send a message that every other device on the bus will receive.

True, but in practice messages that are deemed important are secured at higher OSI levels, and the identity of important bus participants is cryptographically checked during vehicle startup.

> The idea of attaching an internet-connected infotainment computer to the same CAN bus as the brakes is absurd. Doing so on a production car, even more so.

Which is why all vehicle architectures I'm familiar with have a bunch of CAN (and other) buses. Maybe it used to be different, and I certainly know only a few bus architectures, but they make a honest attempt at securing the important bits at least.

Industry engineers may not get everything right, but they're not that stupid. Cut them some slack.

IMO there should be separate infotainment and hi-speed system CAN buses. It's lunacy to place engine control on the same bus as something with internet access that doesn't get upgraded, like, ever. I don't care how well it's tested.

Agreed – FYI: Tesla Model S (and likely X and 3) vehicles use a locked-down gateway (offering specific API actions, which result in CAN communication, etc) between their Ethernet network and CAN [1]

[1] https://www.blackhat.com/docs/us-17/thursday/us-17-Nie-Free-... (page 19), https://blog.lookout.com/hacking-a-tesla

That's how it is on my car, but at the same time my car basically has no fancy electronics.

I understand software engineers pulling stunts like this, because there isn't actually a regulated engineering license, but I'd honestly have expected automotive engineers to be vehemently opposed to putting the stereo on the same system as the brakes.

It's not the software engineers that are the problem it's the automotive design managers who decide that there will be one computer and one buss.

The DOT should have standards for this stuff. Infotainment system shouldn't have control authority or share critical buses with the engine and braking/traction control system.

Sure there is, just not in US where everyone can make cards with software engineers written on them after a bootcamp.

http://www.ordemengenheiros.pt/pt/a-ordem/colegios-e-especia...

https://blog.vdi.de/tag/informatik/

I wonder if some of this is to do with weight reduction. To have seperate buses must add a lot of extra equipment and wiring.

This means more weight and more power required to operate the electrics and in turn means lower MPG, higher fuel consumption and a more expensive car to design and manufacture.

The net outcome is a less competitive car in a competitive market.

I've worked with software engineers working on the CAN bus, and I had the same questions as you. According to them it has to do with the amount of cabling needed as well as electrical interference. A car has a lot of cabling going all over (a couple of hundred meters in total was the number I got), and if you can put as much of it as possible on the same bus(es), you save yourself a lot of problems.

So I suppose weight could be a factor, but space seems to be the big one.

Well the obvious solution it to go wireless! What could possibly go wrong?

It looks like a CAN bus hub weights about a pound. That's negligible, considering I have about twice that weight in garbage sitting in my cup holders.

[1]: https://www.autozone.com/electrical-and-lighting/can-bus-hub...

How much does death weigh?

-21 grams apparently /s

I mean "a lot" in computer terms is like 4 lbs of equipment but since the brake pads on a truck weigh in at more than that I assume its not weight related.

This begs the question: how do you know this is true?

Well JD Power listed Fords MyTouch as the biggest customer complaint of 2011[0].

I think customers might know how messed up their car is.

[0]https://wheels.blogs.nytimes.com/2011/06/23/aggravating-myfo...

My post makes no sense. I replied to the wrong parent

Most modules have some feature or other that needs to communicate with infotainment or another module that needs to communicate with infotainment. Error reporting is the big one, but there's lots of little ones too. The brake module, for example, may be involved in tire pressure sensing, and also takes configuration for traction control and reports activation.

It's hard to find an appropriate segmentation for an air gap without sacrificing features, and most customers won't trade features for "security".

You are right of course. But still. Once you can talk over the CAN bus you can spoof pretty much anything, if I recall correctly. You could at least have some intermediary module on both buses that acts as a proxy. Then if someone hacks your radio they can't pretend to be the ECM.

Then again, this exact scenario actually happened and having an intermediary didn't help. https://www.kaspersky.com/blog/blackhat-jeep-cherokee-hack-e...

The infotainment system should only be allowed to read values from the modules related to the actual driving. Configuring things like traction control will just have to be done via a system separate from the infotainment system.

I understand that it a little wasteful to not use that big touch screen in the center console to do configuration of the driving experience, but there's no safe way to do so.

But the gauge cluster screen is connected to the infotainment system.

An air gap may not be viable, but you can place a firewall there. There is no reason for everything to be on the same bus.

> An air gap may not be viable, but you can place a firewall there.

No idea how it was one or two decades ago, but modern architectures certainly have one.

I can't remember the specifics, but the one I looked at was actually pretty beefy hardware-wise, it had three reasonably powerful cores, 1GB RAM, etc.

> There is no reason for everything to be on the same bus.

It isn't. They would run into bandwidth or (more likely) latency issues anyway.

> IMO there should be separate infotainment and hi-speed system CAN buses.

There are. IIRC BMW uses seven (anyway, several) different CAN buses, plus other bus systems for entertainment (MOST bus)

> Modern electronic cars use a single bus (CAN bus) that connect all electrical systems. The brakes, the engine, the wipers, the stereo...

Not true.

The automotive bus architectures I know first hand (admittedly far from all of them of course) have multiple CAN buses, plus other bus technologies as well (MOST/LIN/Ethernet).

I don't think it would even be possible from a bandwidth/latency standpoint to push everything across a single CAN bus.

I had Hughes as a lecturer in University (Chalmers, Sweden) and we heavily utilized quick check in our Haskell courses. He’s a great lecturer and a brilliant mind, I do think they still teach new CS students Haskell with QuickCheck as a first course.

There should be at least two if not more CAN buses in a modern vehicle. The NHTSA should mandate that safety critical systems be on their own (redundant) network.

Yeah, that's a terrible idea. You should have at least three separate communication channels, and the most critical should be as isolated as possible from the others.

Critical (e.g. brakes, steering, engine control) important (AC, heat, dashboard) and then everything non-essential.

The people that design cars need to be software engineers but instead they're behaving like mechanical engineers.

This seems like just a mind boggingly bad architecture — is there any real reason that a system should be designed this way or is this organization simply the result of poor thinking applied incrementally to inconvenient realities?

well it's real convenient to have all faults available from the obd port, plus it's mandatory in some countries.

The obd port does not preclude multiple can buses and some cars have at least two accessible from the port

In most cars this is not the case. For example, in my 2005 Opel Astra (Saturn Astra for US) there are three buses. High-speed CAN used for critical systems, Low-speed single wire CAN used for other vehicle systems and a mid-speed CAN for the entertainment and climate control stuff. All traffic between these buses are "firewalled" by CAN bridges that should only forward relevant frames between the nets.

VW for example have a firewall in front of the OBD connector, only allowing traffic for the diagnostic addresses to pass.

However, I expect that in newer "cloudy" cars, they need so much data that these "firewalls" have become very permissive. Remote start via Apps, triggering signal horns from the Internet, OnStar telemetry reporting etc.

Traditionally the car makers have been completely terrible at tech security but they are slowly improving on this front. In fairness they've also been to some degree hampered on this front by regulations protecting local small garages, stating that the diag stuff cannot be locked down too hard.

The trend towards monolithic hub systems is baffling. Didn't we learn those lessons while we were struggling with mainframes and timesharing? You don't lock critical systems, ever. You don't even directly interact with them, you send them messages and if they don't receive them they should continue to function as the operator would expect. The collision avoidance system failing to function correctly is no where near as dangerous as arbitrarily applying the brakes.

> Didn't we learn those lessons while we were struggling with mainframes and timesharing?

I feel like there are really three kinds of programmers today: those who learned first-hand the aforementioned lessons, those who were taught second-hand these lessons, and those who are destined to learn these lessons first-hand.

There are very few people relatively who experienced the lessons first hand. They taught the lesson to a larger group, who will hopefully do their best to heed its warning.

But what's happened? A whole lot of people came into programming in the last couple of decades. I feel like there literally aren't enough "elders" in the field to teach the influx of new programming ability. They're making the same mistakes again because for everyone one you tell not to, there are 10 more who are willing to try. And to some of them we're giving millions of dollars and they're forming companies with it, who go on to sell products about which exactly 0 people have asked "Is this a good idea? Has it been tried before? What was the result? What can we learn from that? Has anything changed since then that would lead to a different result?"

Thats pretty much the same story as usenet’s eternal september: as the rate of newcomers increases, it eventually outpaces the rate of teaching. At that point, no culture can be sustained longer than a year; a new batch will arrive, and learn things from scratch, and come up with their own solutions... until the next batch, which overrides any lessons learned by the prior

At the very least, here, the knowledge learned is sustained by individuals, and acknowledged (so you still have “experts” and “elders”), but as a general community, you can safely assume that everything is forgotten very rapidly.

Another example of it I’ve seen in video games, where modern designers seem to have forgotten that games existed before 2005: the same design mistakes solved in 1990 crop up again in 2018.

My preferred example being star control 2 vs mass effect: very similar games in design and spirit, two decades apart, and mass effect features many of the same mistakes, and even mistakes solved in star control. As if they had started the design from scratch.

> and mass effect features many of the same mistakes, and even mistakes solved in star control

I'd be curious to hear examples.

Note that I'm talking about ME1; I never played ME2/3, since I never cared about their combat systems (and it's not very interesting anyways)

Also note that in some fashions ME1 does succeed in its goals, and outdoes SC2, but it doesn't really matter for this; whats interesting is what it failed to learn from its 20 year predecessor

tldr; SC2 does less work and gets much further with it, both mechanically and narratively, in a lot of ways that ME should have learned from. Instead, it seems more like they weren't even aware space operas existed outside of film.

the same mistakes include things like the landing/mining minigame, which is terrible in both games in similar fashion: it's kind of interesting for the first couple rounds until you realize its repetitive, mechanically simple and somewhat poorly controlled. In ME, its a tiny-bit alleviated by being in 3D, so driving around is more fun, but the geology remains dull and generally pointless.

Both feature the nuisance of having to iterate over every world, looking at the stats, and dropping it for 80% as its unlikely to be worth exploring. SC2 also acknowledges its mediocrity and generally avoids requiring it, with very few quests requiring it (and the coordinates usually given), and it becomes mostly unnecessary for mining further into the game as you find massive resource deposits. ME's 3D becomes a negative in this regard, as they still use it for questing throughout the game, and while SC2 has a small 2D screen to parse for the quest location, ME requires "exploring" the dead lands to find whatever marker. The lack of variety becomes more obvious due to the 3D environment (and the amount of time you're stuck in it) compared to SC2.

Notably, the existence of shit exploration in SC2, and how memorable it ends up being, should have been a strong indicator to ME not to pull the same shit

Probably much more based on personal preference, but the much more open, broad and simple narrative of SC2 is more compelling than ME's, at least partially because there's less for them to do a poor job in. ME has bigger and more varied politics between the races, and by far is the more serious game, but this also leads to a lot more stupid politics. Trying to give a bigger background to the races just leads to each race feeling not much different from humans, because, well, they basically are humans in a different skin. Same politics, same motives, different colors. SC2 "cheats" by simplifying races to really distinct, bare-bone traits, and going from there.

The spathi are absurdist cowards, and that's all they are. Their background stories, and all their operations, derive almost entirely from their extreme fear of everything. The Zoq-Fot-Pik are a strange, friendly symbiotic trio of aliens, operating by weird language rules, and odder political preferences. A lot of races lack a clear background, told only in minor hints, and this makes them feel more alien than ME could hope to accomplish. Compounded with the lack of requiring them to walk around and such, primarily being differentiated by their voices and tiny gif-like animations, leaving it to the imagination.

ME makes the mistake that SC2 didn't: aliens are interesting by their very nature; the haze of information is what fuels it. SC2 also has the benefit of being comedic, so it can come up with less plausible stories. ie from SC2's wiki: "For another fifty thousand years, the Zoq, the Fot and the Pik relaxed in the forests, until one day one of each race was walking up a steep path looking for something to eat, when a bolt of lightning struck nearby. The bolt of energy carved a wheel-shaped chunk of granite out of a cliff. As the rock began to roll down the hill, some dry grass got caught in its hole, and since the rock was still hot the grass caught on fire. Thus the Zoq, Fot, and Pik simultaneously discovered the wheel, fire, and religion"

But regardless, it puts out the feeling of Star Trek far better than ME could ever hope to accomplish, at least partially because the game doesn't try as hard. It might be argued that ME was leaning more towards Star Wars (space drama), but even then, it fails by virtue of expunging too much information (and not being grand enough)

And there are other smaller things like SC2 granting a greater degree of freedom, again by virtue of doing less work on its weaker parts. The combat is simpler, and doesn't require it for the most part, and is generally better off for it (ME loves to pretend it has a good enough system; it does not.) SC2 still has the fault of having an annoying amount of combat for what it is, but still far less so than ME (and doesn't really require it for any narrative events). Exploration is more interesting in SC2, again because of the freedom, narrative structure, and it wasting less time on its weaker components.

And of course there are those mistakes that just come out of modern game design, but these are unsurprising:

fast travel: kills any sense of distance and exploration (in ME's case, all travel is fast-travel, so space doesn't feel at all ... distant. The citadel feels larger than space.)

quest-logs: kills any sense of autonomy in the narrative, and background-discovery, and even exploration itself

emphasis on combat missions: ME does not have a good combat system. There's absolutely no reason for it to emphasizing it.

emphasis on player character: It's a space opera, and its focused on upgrading your player character & co? This is just...wrong. Personal preference, but focusing on ship upgrades is much more sensible. But then, the game barely involves space in the first place, so maybe not. It could have been a medieval fantasy and it wouldn't have been too different, in a lot of regards. (ofc, it is bioware, so maybe it really did derive from medieval fantasy)

dialogue wheel: adding morality to the choices, and then making it obvious to the player? It's just absurd. narrative branching on action alone is sufficient, less work, and obviously more correct.

combat-wise there's a lot of mistakes as well, but they're uninteresting for this discussion, and have a lot of other predecessors they failed to learn from. Suffice to say, its not a shitshow, but its not well done. Whats more interesting is that the designers failed to acknowledge its lack of quality, and account for it (or maybe they did so intentionally; its an EA title, so publishers are likely a good deal at fault for that one).

I'd say a bigger problem from my perspective is all the non-programmers who get into management and project management roles who never learned these lessons and who refuse to listen to programmers trying to teach them. They'll ignore you, and if you don't 'be a team player' you'll end up replaced by someone who will.

Because car companies have always built cars, not secure networks. Those executives have never been to DefCon; they have no idea what level of risk they face from the unsecured electronics components in their cars. They are not just unaware that putting critical systems and non-critical systems on the same bus is a bad idea, but they are also completely ignorant of the difference between real-time systems and the operating systems on their phones and computers.

They need to hire some folks with aviation backgrounds, who can explain to them why the plane does not fall out of the sky when the in-flight entertainment system chokes on a scratched DVD. Even when they get it wrong, it is still less wrong than the auto-makers.

[Edit:] Aviation people are the only outsiders they're likely to listen to.

They're not perfect. But they are better. They at least have an awareness that security is an issue, even if the ways they handle it are... well... let's just say they're not ideal.

Even though the tech folks who frequent this site are knowledgeable, the people who build stuff don't always respect our expertise. Sometimes they don't even realize that our expertise might apply to their problems. This is how we get black-box electronic voting machines and CANBUS2 and wi-fi light bulbs or security cameras that inadvertently open a back door into your LAN.

I'd be surprised if aviation systems are much better. They're really into putting everything on the same physical network too, but employing these things called "data diodes". As if that's a real thing, practically speaking.

You can even imagine that Dilbert manager telling you to put those diodes back...

The problem is having virtual 'data diodes', or doing things like vlans on shared complex switches.

A real data diode is easy. You hook up half a serial port.

Sure, but the infotainment system isn't connected via a serial port.

It absolutely could be, though. Or you could use half an ethernet port. The idea of a data diode is fine, it's the [lack of] implementation that's at fault.

I mean, those movies being played on the back of seats aren't being run through a serial port. Given the move to an "on demand" scheme, they're not one way. The data diode is clearly meaningless when there's obviously two way traffic.

I don't understand. You wouldn't store the movies on the avionics systems in the first place. That's all inside the infotainment system, on one side of the diode. The things going over the diode would be stuff like current location and tire pressure.

Ultimately the whole point of having these systems connected on both cars and planes is to save weight on wiring. Their data diodes are just vlans.

Which are often insecure, making them not actually data diodes.

Which is where we started.

And still the problem is not the idea of a data diode, it's implementation.

> "data diodes". As if that's a real thing, practically speaking.

On CAN bus, it actually can be.

I've seen CAN bus participants with the transmit pin not connected. They were physically incapable of writing to the bus (granted, this drastic solution only works in very simple cases).

Maybe we shouldn't use avionics (the article we're commenting on is about a buffer overflow bug in new Boeing planes that could drop them from the sky) engineers to train the auto industry on system security

The other comments have effectively lambasted the manufacturers for their incompotence and foolishness in putting these on the same system, so I'll skip that bit.

The practical reason is that the big central touchscreen is a convenient place to put stuff. On an old truck, you selected 4WD by moving a big transfer case lever, which indicated you were in 4WD by its position. No computer to get in the way. Slightly newer trucks have physical buttons with lights that connect to electrical solenoids, often directly. Tractors have brake pedals for each wheel, but old trucks don't have a similar braking force distribution system - you got locking diffs or dragged all the brakes and hoped it helped. With a touchscreen and a computer in place, it's easy to make compelling sales pitches for putting complex features there instead of on expensive, complicated physical controls.

Complicated on a per-unit basis, that is - I understand that a computer has far more moving parts than any plastic-and-wires assembly, but when it works it's easier to draw buttons and graphics than to make mechanical actuators.

Whether or not the processing happens on the same CPU or communication happens on the same physical bus doesn't matter that much when you have the ability to make selections that affect the engine and drivetrain on the main display.

Makes sense. And it makes sense also to step back a little and see how this design may fail and change it in ways that makes it robust.

Either the engineers thought it was a good idea or the engineers could not convince management that it was a bad idea.

incompetence.

let me explain. attaching a non-essential sub-system with a large attack surface to a mission-critical network is not malicious (I hope) it is incompetence. I would accept a unidirectional serial link with a validated protocol as a sole connection between the two.

I think it is the same kind of thinking that goes into other bad ideas: flickering stop lights, red turn signals. The auto makers just don't really care or think. This is something I don't understand.

It is cheaper.

I've noticed the occasional posting in HN from people who aren't [aspiring] professional software developers, which surprises me.

If you don't mind me asking, what brings someone in your line of work to a site like this?

[edit: I just realized that ^ sounds like a tragically bad pickup line.]

There's also a ton of academic lurkers -- I've always though that this due to the facts that 1) the border between academia and industry is pretty porous in technical fields (super obvious in ML research, but I think holds more generally) 2) academics borrow---and contribute to---the same set of methods and technologies 3) most individuals have an interest in complex systems (and their hilarious, tragic faults) such that they find articles like this one interesting.

I think it's also useful to look at preceding communities like Slashdot — in that case it started out pretty tightly scoped for Linux enthusiasts, gaming, programming, and the then-nascent social internet but then the cohort of regular readers and contributors got older and they found themselves in policy, academia, and managerial business roles; as long as people remained involved in the community, the content of the site expanded to reflect the (extremely rapidly increasing) role of tech in these domains. (Obviously, Slashdot suffered from several ownership changes, a lame commenting system, and most everyone moved on to other venues).

and 4) the signal-to-noise ratio on HN is very good compared to other online discussion forums.

~30% of submissions are software-dev-only (new typescript 3.0 release), but most submissions are just high quality, interesting posts. And the level of discussion is excellent - no memes, no jokes. Its like the opposite of reddit.

Yeah this. Signal:Noise ratio on HN is far better than any other source of tech news & discussion I know of. I skip the programming-related threads.

I think Lobste.rs might sneak past HN on signal:noise, but it gives up a bit of variety in topics.

our heavy diesel emissions performance machine runs Linux, and after a vendor training on how to use it I became totally fascinated with it. A lot of Linux professionals like HN a lot, and the articles are a good break from the norm on lunch break.

Im learning python, too! :)

I've been lurking here for 10 years. For most of that time I was employed as an electrical engineer.

Why should people who are interested in high quality technical discussion surprise you with their presence?

"Hacker" used to mean a lot more than "software developer".

I came across HN by accident, and what keeps me coming back is the interesting, informative links & threads, and the quality of discussion. Threads about workplace issues are of interest to me, as are general scientific and social topics.

I'm a "scientific" programmer, meaning that I use programming as a problem solving tool, but I don't write commercial software for a living. I turned away from that path in the early 80s.

I've been reading this site since 2009 and can barely write anything more complex than Hello World. The guidelines on submissions say nothing about software engineering, and the definition of hackers as written by Paul Graham does not limit it to software engineers.

The comments are informative, there's no politics, it's more interesting than the news. I'm a sales recruiter, so I don't read the in the weeds software articles, but there are still 2-3 good articles on the front page at any one time.

> there's no politics

Ok I am definitely clicking on bad articles

HN is not a monoculture the appropriate stuff on the site can interested anyone and no one is required to be a software developer to get on this website.

Nobody is required to be anything on any public forum; that doesn’t change the fact that communities exist, and the vast majority of people on a forum are there due a particular taste, and share a common preference for discussion.

That anyone of any interest can visit the webpage is not a sufficient explanation as to why anyone of any interest do visit it. The question is whats so special about HN that draws such a varied background, despite it clearly having its strongest preference towards software engineering (on any given day, if you counted by category, it would be surprising to find software engineering / programming not as the highest).

A reddit forum has the same public property, but the software engineering channels presumably do not draw such varied backgrounds. So why?

Maybe granularity -- there's only one front page for HN? I usually find that I want to read 10% of articles on the front page, but I'm often surprised by which 10%.

Most likely to the type and extent of moderation that is present on HN and the evolving purpose of both websites .

A lot of the content is still relevant for those in other IT roles, or even just hobbyists. I'm a security engineer rather than a dev but I still find many of the posts on the site useful and/or interesting

Big factor for me is the higher-quality discussion compared to say, Reddit.

Not everyone in the software world is a developer.

you had me at "bad pickup" :)

That's pretty spectacular, I'd love to read some more sources on it.

Makes me glad Tesla did their stuff the way they did. You can reboot the cenelter console and/or days while driving and not much happens other than A/C turning off for about 20s.

Tesla pushed out a software update to AutoPilot (Lane Centering) that caused a fatality[0], I hardly think they should be held up as an example of good software hygiene.

[0] https://www.mercurynews.com/2018/03/29/tesla-crash-victims-f...

There was recently an interesting twitter dump by a former Tesla engineer whose NDA was up; as of a few years ago their software development practices were... pretty bad.

Apparently every Tesla vehicle runs a Kubernetes cluster and is/was remotely accessible to Tesla via SSH?

https://twitter.com/atomicthumbs/status/1032939617404645376?...

I sleep very well at night knowing my software is not keeping helicopters in the air, cars on the road, etc.

Honestly I'm in awe of the lot of programmers who take these kinds of tasks on. I don't know how they can relax at all relative to their work.

> I don't know how they can relax at all relative to their work.

They strictly follow procedures. If faulty code gets through to production and somehow causes a fatality, it's a failure of the procedures, not the individual developer. This works so well in aviation and yet seems completely non-existent in the automotive world.

The procedures exist and are 'followed', but there's a lot more time-to-market pressure. Manufacturing lines dictate the schedule and software must keep up no matter what.

When I worked on safety-critical products, it was actually really nice to be able to push back.

"We can't satisfy the safety case yet" is all you need to say to get your manager to cave.

If they want to take the risk upon themselves to sign off on a known-unsafe (technically; in practice it was already pretty good, just not good enough yet) device, and go to jail if something goes sideways, they can be my guest...

In practice, they preferred to come back next week and ask if we were done yet.

The project went wayyyy past the deadline, thanks for asking :-)

If you as an individual are taking on that level of personal responsibility, your organization is broken.

Part of the regulatory process is to demonstrate sufficient levels of resilience in both the systems produced and the development process used to produce them. It should be a really boring process with a lot of crosschecks. You should be sleeping well knowing that your work is being reviewed and tested and attacked by other people.

(Reality is often different, of course. Just don't try to be the hero.)

I recommend reading the "NASA Manager's Handbook for Software Development". It contains some great guidelines for writing safety-critical software. AFAIK, no Space Shuttle mission ever suffered a serious safety incident due to a software defect.

https://ntrs.nasa.gov/search.jsp?R=19910006460

> I sleep very well at night knowing my software is not keeping helicopters in the air, cars on the road, etc.

Hopefully at least, but you never know where those random shell scripts you put in a gist (or whatever) might end up.

I'm with you, I'm nowhere near confident enough in myself to work directly on safety-critical systems. I imagine an important part of those organizations is removing SPOFs with respect to the engineering team itself; i.e. nobody fails alone, and everybody tests the hell out of everything. Could be wrong though.

If you work for AirBnb your bad software is definitely changing lives and putting people in danger. Software doesn't have to be embedded to materially affect someone's life.

Saying it caused "a fatality" is being disingenuous. It's along the same false pretense as "we shouldn't use self driving cars at all, if they lead to even one fatality".

Fatalities aren't good things, but they're inevitable consequences of driving. And it's disingenuous to expect anyone's code to be guaranteed to work on every mile of road in the US, under all conditions, at all times. If autonomous vehicles substantially reduce overall fatalities we are better off.

(Nobody talked about banning combustion vehicles after the pinto or banning Fords over their side mounted fuel tanks or their explorers that rolled over when the defective tires blew out)

> Saying it caused "a fatality" is being disingenuous.

It is accurate.

They pushed out an update that changed how AutoPilot behaved, someone had AutoPilot enabled, AutoPilot accelerated straight into a concrete barrier, and the occupant died.

Here is the NTSB's initial report on the incident:

https://www.ntsb.gov/news/press-releases/Pages/nr20180607.as...

> It's along the same false pretense as "we shouldn't use self driving cars at all, if they lead to even one fatality".

I didn't say anything remotely like that. I stated what had already occurred. Trying to put those words in my mouth doesn't seem like you're responding in good faith.

You have no proof that the accident wouldn't have occurred anyway without the update. Or that their updates didn't save one or more lives.

That's my point. Not an "accurate" but misleading accusation at the software without considering the hidden variables.

"You have no proof that the accident wouldn't have occurred anyway without the update."

Well, the NTSB preliminary report doesn't make any statements about what the software update did, per se. But it does indicate that the driver's hands were not on the wheel at the time of the accident and that the "autopilot" software was activated. So it's fair to say that the "autopilot" software was responsible for the crash.

"Or that their updates didn't save one or more lives."

This is a red herring. It could be true. But there's no evidence for it so it's not worth thinking about. Our system of moral judgements rightly puts a lot of weight on demonstrable causes and effects, and generally ignores hypothetical, but totally unproven speculation like this. Otherwise anyone could get off the hook for anything, by saying, "I may have done bad action X but you can't prove I didn't also do good actions Y and Z which could well outweigh X."

"Not an "accurate" but misleading accusation at the software without considering the hidden variables."

In ordinary life we never know all the hidden variables. We just make judgments based on the best available information (or if necessary, decide to postpone judgment until better information is available). The NTSB preliminary report seems credible and I see no reason not to draw reasonable conclusions from it.

Tesla really scares me.

They released an OTA update to the brakes with less than 2 weeks from first reports to code being pushed to the entire fleet of cars on public roads.

Toyota had a similar brake software issue in 2010 [1]. They had a fix available 3 days after the NHTSA announced the issue. Toyota issued a voluntary recall for over 130,000 cars in the US. Since the recall was voluntary, it likely took years until some cars received that update when they were serviced for other unrelated issues. I have no idea if Tesla is doing things the best way, but Toyota having the ability to push that update out immediately would have indisputably prevented accidents and possibly saved lives.

[1] - https://en.wikipedia.org/wiki/2009%E2%80%9311_Toyota_vehicle...

Move fast on brake things?

I’m very glad that Tesla’s instrument cluster is semi-independent. Because mine crashes every few minutes while driving due to a “high priority” firmware regression.

On the latest ADB Podcast, they mention that certain car infotaiment systems just go "blue screen" if you feed them a more recent Bluetooth version than what they have hardcoded.

http://androidbackstage.blogspot.com/2018/08/episode-97-blue...

John Hughes likes to talk about the CAN bus when discussing Quickcheck and test generation.

Learning about the crazy complexity and potential for unforeseen interactions made me very unhappy when security researchers remotely controlled a car on the highway as a publicity stunt. There’s simply no way they can know with absolute certainty what might happen.

Yikes. I'm not a prepper, but there is something to be said for older vehicles without this unnecessary computer nonsense.

Question from the ignorant:

What % of a 24-hour day is any given commercial airliner in service, vs. sitting completely turned off in a "parking" spot?

Most modern things don't ever "turn off".

Uhh that's kind of terrifying. I'm due for a new car after 10 years, and I think I might specifically try to avoid the systems that centralize things like breaks with entertainment systems...

That’s horrifying. Are there any comprehensive write ups on this you can share? I’d like to read more about this.

This is just ... outlandish. You couldn't make this shit up with a straight face.

Why aren't these systems just plain air gapped?