I think this is a naive perspective. I've heard many similar stories from friends that work in the German automotive industry, and I think you would be surprised how many payment systems are tied together (e.g. (insecure) FTP servers to sync daily payments).

Every organization has these types of things internally. As an engineer, I don't like it, but it's a fact of life.

Banks can reverse transactions if they want to. Tesla's cavalier attitude towards manufacturing has yielded a 14% first pass through rate on the Model 3 line (which is abysmal and may bankrupt the company along with declining M3 demand) vs. the industry standard of ~80% FPT rate.

Source - https://www.businessinsider.com/tesla-hit-model-3-target-by-...

We're at the point in the Tesla story similar to that of 2008 where Dr. Berry et el were watching the housing market collapse around them and the banks wouldn't re-price the swaps. Tesla is already bankrupt, most people don't know this yet. But they will.

I don't think the rework rate is a big deal for the customer. Ultimately manufacturing is about increasing the yield rate so that costs are lowered, because rework is expensive. But if you can make money with lots of manual rework on every product, it's no big deal. Something to improve next quarter.

I have so many electronic devices, from cheap to expensive, that have some passive component manually bodged on somewhere. They work fine. It just means that paying someone to rework 1000 boards was cheaper than throwing the boards away and spinning up Revision B immediately. It's not a big deal. Waste is worse than a product that is imperfect immediately off the assembly line.

Rework dramatically slows production.

Tesla cannot get the rest of their booked-sale cash for an undelivered car.

But they must pay all the fixed production costs anyway. In cash, creditors at this point will be getting squirrelly.

Therefore: slowness in production is the fast-track to a possibly fatal cash crunch.

People are waiting 3 or 4 months for replacement parts for their vehicles or Tesla has had possession of the vehicle (and the person is using a loner) for a similar amount of time. That won't fly in the mass market.

"But if you can make money with lots of manual rework on every product, it's no big deal. Something to improve next quarter."

Is Tesla making money?

>>> It just means that paying someone to rework 1000 boards was cheaper than throwing the boards away and spinning up Revision B immediately. It's not a big deal.

It's not about money. Correcting and refacturing a board takes time, in the order of months even for a simple one. It's too long.

That link is not a source for declining Model 3 demand. Source? My info shows that

• In July, over 60,000 test drive requests in the US alone

• 5000 Model 3 new net orders in one week in mid-July

• Total deposits greater than total refunds in the twelve months ending April 2018

• Reports of Cancellations Outpacing Orders proved to be False wording

Meaningless negative indicators include Goldman Sachs analyst David Tamberrino saying Model 3 social media activity had lessened and frequent critic Latrilife said Tesla’s Burbank Airport lot is under 24/7 surveillance

Sources include:

• Seekingalpha.com/article/4189303-tesla-short-thesis-model-3-demand-wrong

• Seekingalpha.com/article/4179986-tesla-model-3-reservations-tip-iceberg

A key supplier may have gone cash on delivery (no more net 120 terms) because M3 production this week has been at a stand still - https://twitter.com/skabooshka/status/1032409945407250434

And - https://twitter.com/Paul91701736/status/1033136573615730688

All the first hand Twitter intelligence is gathered by people who are short on TSLA. They see the ground level reality.

While I agree that 14% is horrible, that was a point-in-time number that is being compared to an average. One would hope that factories regularly run well above 80%. I'd also bet that some occasionally drop well below it for a day or so.

The details matter here and given the quality that I see in the field, I'm not convinced that this is such a horrible situation.

In some cases banks can't reverse transactions.

>I think you would be surprised how many payment systems are tied together (e.g. (insecure) FTP servers to sync daily payments).

It's not like those matter. The bank itself guarantees the integrity of your account and can reverse charges. And of course would be insured for such losses.

Being killed by a car, otoh, is more permanent.

Tesla’s infotainment and IT infrastructure is unrelated to their safety. If this guy worked on motor control or braking system firmware then that would be scary, but he didn’t.

If the infotainment system caused the MCUs to reboot while someone traveling "130mph on San Mateo Bridge" and that caused the break system to segfault due to unconventional way of loading parts firmware, it might be a life&death situation, easily. Examples in that threads go on, literally hundreds!

Well, you can reboot the system while driving(both console and dash), nothing special happens other than the AC turning off for a brief period of time. Brakes, wheel, throttle all respond normally.

Source: Done this a few times to clear bad map data or occasional glitch.

Doesn't surprise me -- a couple of times my new 3 has had nothing on the display but it's still happy to let me put it in gear and drive away, and the display pops up within a second or two.

To be sure, it's a bit unsettling, and I wouldn't be thrilled about it deciding to not work for a day or two, but, in a way, it makes me MORE comfortable that the vehicle control systems function as expected.

AC should be considered mission critical.

Definitely, I'd put it on the mission criticality list right below inflight wifi.

It's not mission critical, but it's important: in cold weather the windshield can fog up to the point of 0% visibility.

In hot weather it can become very uncomfortable or even impossible to drive without AC.

There’s a distinction between safety critical equipment and essential equipment. If the former fails, it could kill you. If the latter fails, you can’t drive anymore but you won’t die if it happens on the road. Brakes are in the former category, while things like HVAC and instruments are in the latter.

Safety equipment must not fail, but essential equipment can fail as much as your customers will tolerate.

You can steal a trick the motorcycle community discovered a long time ago, some water + soap on the windshield prevents condensation.

Our '83 Vanagon doesn't have AC, it can be uncomfortable but sure isn't the end of the world.

Seriously yes, at least ventilation. Anybody who don't believe this, try turn off your AC completely and see how quickly your wind screen is fogged up to the point you can't see out. (depending on which climate you live in)

I don't understand how that could happen. MCU reboots, ???, brake system segfaults? What goes in the middle?

Probably everything has access to everything on the CAN bus.

IIRC car manufactures are irresponsibly lazy and never properly air-gap infotainment from critical stuff.

Even if this were true (and as another commenter says, it's not) I don't see how that fills in this gap. It would make sense if the first step were "MCU goes crazy," but not with a simple reboot.

There's a firewall between the CAN bus so it cannot be affected at all by the MCU.

There’s no concept of “reboot” with MCUs, since there’s usually no OS. Likewise there’s usually no concept of segfault, because segfault requires memory protection which is something most MCUs don’t use.

Tesla’s MCU runs Linux. Older ones use a 2012-era Nvidia mobile SoC, while newer ones use something from Intel.

Nah. No way the mcu responsible for braking would run an OS

This must be an acronym mixup. We’re talking about the Media Control Unit, i.e. the giant screen in the center of the dashboard responsible for zero safety-critical systems.

Yes, The infotainment stuff, like the screen, internal lights, speakers are connected via a low profile third bus system, certainly not the main CAN bus or profibus or firewire. Some, like in the BMW even via WiFi. These are the systems usually used with Linux or Windows or Android. On top of the important stuff.

MCU in the case of braking and steering is “micro controller unit”.

Yeah, those definitely aren’t running Linux, and the guy here wasn’t working on them.

Didn't they already do an Over-the-Air update to fix the braking time in response to a bad review?

You don’t need an OS for that.

s/reboot/reset/, s/segfault/bus-fault/

Ignoring the infotainment system for this argument (as they firewall it off from CANBUS and other life critical systems [1]), I argue that their IT infrastructure is safety related, as it governs Tesla's velocity in getting patches and security fixes out to vehicles in a timely manner.

Can you imagine a zero day being found in Windows with Windows Update being down?

[1] https://www.youtube.com/watch?v=KX_0c9R4Fng

There's no guarantee that any given car is connected and receives updates, so the safety-critical systems need to be good enough when the car ships. They might mess up, but then they'd at least be able to patch cars faster, while other manufacturers would have to do a recall.

1. According to this source, the infotainment system arbitrates flashing other ECUs.

2. Apparently people can ssh into the infotainment system.

1 + 2 => someone hacks into the infotainment system and flashes a safety-critical ECU. I guess/hope there's some protection in place to prevent that.

> Tesla’s infotainment and IT infrastructure is unrelated to their safety.

Only if it's deliberately isolated in the vehicle. It should be. Aaaand, it isn't: the firmware upgrades to all other car computing elements go through it.

Runtime isolation is distinct from compile/build-time isolation. You're citing the latter, but it's the former that matters. Tesla gets this right, e.g. an interrupt in the MCU does not have any effect on braking, drive-by-wire, or ADAS systems while a car is in operation.

Just to back this up: I have rebooted both my MCU and my instrument panel while driving. Critical systems are not affected.

Fair enough, but if those bits pass through the infotainment center on the way in, controlling one allows you to control the other.

Well, the model 3 lacks a normal speedometer, it's in the screen. For that car, it's related to safety.

>Tesla’s infotainment and IT infrastructure is unrelated to their safety

Because drivers can't get distracted and crash because of a failure in the infotainment and IT infrastructure?

If that's the standard, then almost everything becomes safety critical. Drivers can easily get distracted by malfunctioning smartphones or apps. (Or, for that matter, properly functioning smartphones or apps.) Yet the prior discussion was based on the idea that things like iPhones and Facebook aren't safety critical the way this is.

>If that's the standard, then almost everything becomes safety critical.

Almost everything in a cars front panel and dashboard can be. I've read somewhere that people have been killed even because of something as quaint as the wrong placement of car ashtray.

One time I pulled up at the red light behind a Tesla while I was on a bicycle. Straight through the rear window I could see the driver and front passenger being distracted by the massive flat touch screen. The traffic light turned green, and had I been beside the Tesla, I would have beaten it across the intersection despite all that electric motor tech in the car.

Anyway this little illustrated anecdote of mine aside, driver distraction is a genuine issue - even for drivers at red lights. The number of times I've seen this type of behaviour transcend into moving off while remaining distracted (whether that's heads down visually or just mentally) is too boringly frequent to detail. Sometimes the distracted drivers even creep forward unconsciously while traffic is flowing across them. Emergency vehicles can't get through, drivers end up splitting their attention, pressure mounts once proper movement starts again, and all the while they don't realise they don't have full attention in a changing environment.

It's almost like a hypoxia.

Tom Scott did a pretty good video on this: https://youtu.be/_-aDHxoblr4

I've missed the light turning green while just looking out the window (our car has no screen). As long as people are only distracted while waiting at a light I wouldn't worry too much.

Except they're not only distracted while looking at their screen when stopped. The distraction continues after they start rolling again - one thing directly leads to another here.

It is my understanding that two major reasons for various ugly UI and unintuitive UX in automotive infotainment systems are patents and safety certifications.

The iPhone has to reliably connect for emergency calls and provide accurate location. That can be life or death. In fact, "Apple isn't ready to engineer phones at a life-or-death standard" was a fairly common critique of the iPhone in the early days. In response, Apple ran a little PR campaign around their radio engineering efforts--they had a web page that showed all the cool-looking rooms for testing radios, which they invited a couple reporters to tour, etc.

> The iPhone has to reliably connect for emergency calls and provide accurate location.

An iPhone handling an emergency call is an exceptional use case.

A car driving is not an exceptional use case for a car.

A life and death scenario is occurs every few minutes, or continuously for something like a mountain road, if something like brakes, limited throttle, or steering were to fail.

But, I would, naively, assume that the operation of these critical components were in no way related to timing requirements of some process in Linux.

Is a Tesla steer-by-wire? No.

It seems you have to prove that failure of these hack-y systems would lead to the disastrous result you postulate.

Brakes aren’t either. You might lose ABS or boost, but you’ll still have brakes.

That PR tour was in response to "AntennaGate" when Apple failed to engineer a phone to a life-or-death standard (well, if you were "holding it wrong").

Didn't the iPhone use a dual processor set up like the other manufacturers at the time? One ui/app processor and a separate radio processor stack?

Evert smartphone separates the baseband processor from the frontend android/ios/... Arm processor.

Even PC's have now a hidden baseband processor for the mission critical stuff, like backdoors and surveillance.

There's a lot of EE black magic around antenna design.

Are there any interesting papers on formal verification of some of the most modern machine learning algorithms?

Clearly when we "verify" Waymo or Tesla auto-pilot, we're going to want to use that stuff, right? Surely they won't just provide insurers with some data about the billions of miles they've driven without accidents and how humans can only drive like a million miles without an accident and try to get the insurers to give them policies...

Just like when we hand out licenses, we always check to make sure the 16 year old took some formal driving classes from professional driving instructors... I wish things were better but we don't care about this stuff as a society until much later usually. When did the car first appear? When did the first seatbelt law appear?

lol they'll never bother with that

Read the Barr report on Toyota’s unintended acceleration incident. They didn’t have the right failsafes and watchdogs and only a single bit was responsible for a critical feature. Meaning memory corruption or a cosmic ray flipping the bit can cause disaster. They didn’t follow anywhere near best practices in their firmware development. It’s not just the young upstarts that mess this up by being “reckless”

There's still the problem with exploding phones and batteries so even if it's just a phone it can go wrong in dangerous ways. (If it happens on a plane for example)

Indeed, cars currently move fast and break things. Tesla is an improvement on current situation.