News for hackers. Stuff that matters.
Re:A recent software update was not successful. Your vehicle... by otterley on Monday December 25, 2023 @04:30PM (#38766304)
It’s a bit troubling that automakers (and, well, anyone who is building things that have mechanisms for software updates) aren’t making software updates failsafe. If you have multiple copies of the image on storage, and you take care not to touch the currently-running image, then a failed software update to the alternate image shouldn’t have any impact. Think of it like blue-green deployments on the cloud; and it’s what TiVo was doing with software updates to their DVRs all the way back in the late 1990s.
Re:A recent software update was not successful. Your vehicle... by variadix on Monday December 25, 2023 @04:35PM (#38766347)
My router has dual firmware partitions, the fact a car doesn’t is inexcusable
Re:A recent software update was not successful. Your vehicle... by g_p on Monday December 25, 2023 @06:33PM (#38767373)
Absolutely. One fact I recall from many years ago (so I don't know how true it is now) is that auto makers were shipping 7 year old silicon (and thus board support packages for their software). The guys working on chips and CPUs were writing software and it wouldn't hit the road for 7 years.

This was why infotainment systems lagged the state of the art for so long - they were apparently shipping components that would have belonged in a smartphone from more than half a decade before.

In this case, I believe we had a few phones get A/B updates by Android 7, which was late 2016. Giving it a year or so to filter to more CPU chipsets, that would mean (if the old rule held true), we would expect to see A/B hitting new cars in the next year or two. This assumes auto makers are using enough of the Android BSP to get the bootloader with A/B (which isn't a given), and then that they use it!

Anyone with any experience of auto software know if the teams doing software are experienced and understand the platform side of things, or if it's a traditional front-end heavy "crunch release" type affair? I've seen some pretty poor quality infotainment systems in the past, so my assumption is inexperienced devs with limited platforms knowledge, focused on "get it working to ship it", but would be interesting to hear from anyone with experience of how this really goes down.

Re:A recent software update was not successful. Your vehicle... by justsomehnguy on Monday December 25, 2023 @07:24PM (#38767726)
It was a great advantage, both in marketing and the real usage, when the motherboards started to support the Dual BIOS (GIGABYTE term, AFAIR). 2003 I think?
Re:A recent software update was not successful. Your vehicle... by KennyBlanken on Monday December 25, 2023 @05:16PM (#38766700)
It should be mandated, but corporations dominate legislation and regulations.

"You'll drive up costs for the hard-working American consumer"

"We'd have to cut salaries or fire factory workers"

"We wouldn't be able to compete against that car company from Nothereistan!"

etc.

Re:A recent software update was not successful. Your vehicle... by cedws on Monday December 25, 2023 @04:46PM (#38766440)
This is more commonly known as A/B partitioning and as you say, is standard practice in embedded devices. It is indeed disturbing if automakers have not adopted this, nor a rollback mechanism.
Re:A recent software update was not successful. Your vehicle... by amelius on Monday December 25, 2023 @05:51PM (#38767049)
We should have a law for this. In most countries, new vehicles need to be tested by govt agencies before they are allowed to hit the road. The same should hold after software updates.

Ideally, the update should be done by independent agencies, so they can make sure the software is tested and they can properly inform the user (something which we already know companies don't always do).

The way it is now, it is all too easy for a company to decide to push a quick fix for a problem that really needs a more careful approach (but which might cost the company money).

Re:A recent software update was not successful. Your vehicle... by andsoitis on Monday December 25, 2023 @05:56PM (#38767097)
You want an agency to validate every single software updates to vehicles?
Re:A recent software update was not successful. Your vehicle... by norir on Monday December 25, 2023 @06:20PM (#38767289)
I actually think this isn't a bad idea. I really don't want an automaker treating consumer updates like a developer repeatedly pushing nearly empty commits to github to get ci green.
Re:A recent software update was not successful. Your vehicle... by amelius on Monday December 25, 2023 @06:00PM (#38767133)
At the very least, I want the agency to keep a hold on the update for $N days.

That way, I am more convinced that the software engineers of the car manufacturer won't push any quick fixes that might cause bigger problems later.

In the case of urgent problems, the correct response is to ground the fleet. This will cost the manufacturer money, but I'm a software engineer for long enough to know that a quick fix won't be a safer solution.

Re:A recent software update was not successful. Your vehicle... by throwaway2037 on Tuesday December 26, 2023 @01:35AM (#38769391)
> At the very least, I want the agency to keep a hold on the update for $N days.

What is the difference between this and "No Deploy Fridays"?

Re:A recent software update was not successful. Your vehicle... by Klonoar on Monday December 25, 2023 @06:51PM (#38767472)
At the very least, if it touches anything that impacts the driving… maybe?

It’s a 3 ton (+) machine with people paying attention less each passing day. It should be taken very seriously.

Re:A recent software update was not successful. Your vehicle... by andsoitis on Monday December 25, 2023 @09:53PM (#38768547)
A scalable way, and which I believe is in place, is having rules and then periodic audits validates that the controls and processes provide a high degree of assurance of compliance with the rules and regulations.

This seems far better than having an agency ad hoc review every single specific software update to vehicles - would people at the agency review the code? Do they write automation? Do they do manual verification?

Re:A recent software update was not successful. Your vehicle... by iknowstuff on Monday December 25, 2023 @04:52PM (#38766488)
Tesla is. Failed updates are non-destructive, they also have an RTOS gateway isolating entertainment software from the software needed to drive the vehicle.

https://www.pentestpartners.com/security-blog/reverse-engine...

And this is their decade old architecture. If you watch some cybertruck videos you’ll see it is much more elegant now.

Re:A recent software update was not successful. Your vehicle... by KennyBlanken on Monday December 25, 2023 @05:10PM (#38766627)
This isn't a tesla-specific thing or even something they innovated. The earliest cars with data networks had this.

It's also not something Tesla does better; numerous security researchers have bypassed that gateway.

Since you bring up Tesla: used to be that Tesla control units would fail after a certain number of drive time hours because they did so much logging to the flash chips they'd wear them out.

In a vehicle which is highly connected and basically should never lose power most of its life, they were writing logs to flash memory. Anyone who has done embedded work can tell you how stupid an idea that is.

Re:A recent software update was not successful. Your vehicle... by xw3089 on Monday December 25, 2023 @05:45PM (#38766997)
Yes, that happened to my car. But one could still drive their car after the failure and there was a free recall.

https://www.tesla.com/support/8gb-emmc-recall-frequently-ask...

Re:A recent software update was not successful. Your vehicle... by stwr on Tuesday December 26, 2023 @05:58AM (#38770674)
Genuine question, in that scenario what should you write the logs to if you wanted to keep the files locally on the car?

I understand EMMC/Flash is a bad idea, but what is the better approach, a HDD/SSD?

Re:A recent software update was not successful. Your vehicle... by rpmisms on Monday December 25, 2023 @05:01PM (#38766549)
Tesla is very very good at software engineering. They still need to find the balance in their software and process engineering, though.
Re:A recent software update was not successful. Your vehicle... by stefan_ on Monday December 25, 2023 @05:27PM (#38766804)
You are talking about the company that wrote logs to the fully read-write filesystem of their infotainment system and killed the EMMC from raw flash cycles within as few as 2 years. Recalled 10 years later.
Re:A recent software update was not successful. Your vehicle... by rpmisms on Monday December 25, 2023 @05:31PM (#38766846)
Yes, this happened to me, although it took 10 years. Free replacement, and the car still worked fine.
Re:A recent software update was not successful. Your vehicle... by FirmwareBurner on Monday December 25, 2023 @05:50PM (#38767043)
It's still a major embedded engineering oversight on their part. We had fallback developed for such cases in $5 embedded devices.

Calling out Tesla as a model for excellent software is insulting to SW that's actually quality.

What Tesla SW has and others lack, is a great "wow" factor, which consumers who don't get to see how the sausage is made, mistakenly associate with 'quality' because it behaves like their iPad, but don't know all the skeletons in that closet.

Re:A recent software update was not successful. Your vehicle... by rpmisms on Monday December 25, 2023 @07:00PM (#38767549)
Yeah, it's also a mistake they made 10 years ago.

Tesla is very good at automotive software engineering.

Re:A recent software update was not successful. Your vehicle... by FirmwareBurner on Monday December 25, 2023 @07:11PM (#38767635)
>Yeah, it's also a mistake they made 10 years ago.

You're making it sound like that mistake was an "I spilled my coffee on you boo-boo" and not a terrible engineering decision that should not have passed anyone with actual experience in the embedded industry, let alone a team. If this is the kind of engineering oversights that ends up in Tesla SW, only God knows what else they have lurking in there that hasn't yet reared its ugly head.

>Tesla is very good at automotive software engineering.

Citation needed. Repeating something doesn't make it true.

Re:A recent software update was not successful. Your vehicle... by ahahahahah on Tuesday December 26, 2023 @03:20AM (#38769838)
Yeah, they have some truly breathtaking software controlling the automatic wipers. It's like a true ai, probably competitive with a 6 month or so old child. The automatic high beams are up there with the pinnacle of software engineering as well.
Re:A recent software update was not successful. Your vehicle... by rpmisms on Tuesday December 26, 2023 @08:21AM (#38771535)
I've heard a lot about how bad they are, but I've just never noticed either doing anything but their job.
Re:A recent software update was not successful. Your vehicle... by superkuh on Monday December 25, 2023 @05:09PM (#38766618)
Anyone on the 'net back in the 1990s will remember the image macro, email forwards, and usenet posts about "If cars were made by software companies". It looks like they're finally coming true.
Re:A recent software update was not successful. Your vehicle... by soganess on Monday December 25, 2023 @06:12PM (#38767234)
Except the price part. We have an inverse Miley Cyrus situation on our hands.
Re:A recent software update was not successful. Your vehicle... by j45 on Monday December 25, 2023 @04:57PM (#38766517)
It's really interesting that updates can't revert, or enable some limp mode, or disable the offending functionality.
Re:A recent software update was not successful. Your vehicle... by stefan_ on Monday December 25, 2023 @04:45PM (#38766436)
I'm sure the infotainment system does, but that is not all that is being updated; you have many more processors in ECU, assistance systems and so on that are also sometimes getting fed new firmware over various wonky buses as part of an upgrade. In some instances there might even be multiple levels involved in the firmware distribution.
Re:A recent software update was not successful. Your vehicle... by yjftsjthsd-h on Monday December 25, 2023 @05:18PM (#38766720)
So? Everything that can update firmware should do the same thing.
Re:A recent software update was not successful. Your vehicle... by stefan_ on Monday December 25, 2023 @05:25PM (#38766787)
And then what? A/B update isn't magic dust, if some sub component ends up with an incompatible firmware because it had to boot to the alternate partition, you might still have an incompatible overall system.

(Not like the subsubsupplier of that module even specced the extra flash memory for A/B update, hah)

Re:A recent software update was not successful. Your vehicle... by mistercheph on Monday December 25, 2023 @07:18PM (#38767686)
Retain the known good firmware for each controller on master disk A, if the update fails, reflash all components.
Re:A recent software update was not successful. Your vehicle... by gpderetta on Monday December 25, 2023 @06:12PM (#38767236)
The real question is why would they be doing OTA upgrades of such critical systems.