News for hackers. Stuff that matters.
Re:Rivian software update bricks infotainment system, fix no... by mlyle on Tuesday November 14, 2023 @12:44PM (#38266693)
The code went through early release tests successfully; the problem came with how it was more broadly released.

They should have had further staging of the rollout (randomizing when it is offered to users).

Re:Rivian software update bricks infotainment system, fix no... by whalesalad on Tuesday November 14, 2023 @12:54PM (#38266844)
A/B partitions tends to solve that. You will only switch to the new partition when the update is 100% verified installed. If it doesn't complete in an atomic manner, your device will just boot into the previous healthy partition.
Re:Rivian software update bricks infotainment system, fix no... by AlotOfReading on Tuesday November 14, 2023 @01:06PM (#38267040)
A/B gets complicated in the real world. BL1 may not support A/B for example, so to implement A/B bootloaders you may need a shim that can read/write NVM to handle that. Your HSM may not have slots for multiple keys to have different signatures, so upgrading one may trample the other if your update code doesn't check that.

Lots of ways to screw this up, especially in automotive where you're likely to be dealing with TI and their (in)secure boot.

I've solved this problem god only knows how many times now and I've rarely found an automotive board that doesn't introduce fun, new edge cases. OTA can't exceed x kilobytes of memory, the processor isn't fast enough to verify signatures and write the image in < x seconds, can't write the image to flash unless the signature is verified, but the image doesn't fit in RAM, the server delivering the update is 3+ networks away from the device receiving the update, etc.

Re:Rivian software update bricks infotainment system, fix no... by Dalewyn on Tuesday November 14, 2023 @01:17PM (#38267201)
With all due respect, that all sounds like Programmer Induced Problems(tm).

Cars are a long solved problem, being around for over a century. Telephony and computing hardware and infrastructure today are in the realm of the ludicrously good compared to even just a few decades ago, even if we consider bottom of the barrel worst case scenarios. If software somehow can't work a solved problem using ludicrously good hardware, the programmers (and their managers) are the problem.

Re:Rivian software update bricks infotainment system, fix no... by whalesalad on Tuesday November 14, 2023 @01:56PM (#38267795)
I agree here. A partition is just a partition. It's taking one disk and abstracting it into two. This is not hard.

The rovers on Mars, the Voyager, these problems have been solved for a long time. The compute in a Tesla these days can probably run Crysis.

You can do this OTA to a Raspberry Pi running Nerves via remote SSH and it works really well. The Nerves runtime utilizes A/B partitions for OTA updates.

Re:Rivian software update bricks infotainment system, fix no... by mypalmike on Tuesday November 14, 2023 @02:29PM (#38268298)
I once worked at a startup that sold a very expensive "enterprise" network appliance that didn't have partitioning. I had worked on network appliances before which did have this capability, so I asked the VP of engineering about it. He said it just wasn't a priority given all the other work that needed to be done. I wouldn't be surprised if Rivian had the same startup mentality that might lead to such a situation.
Re:Rivian software update bricks infotainment system, fix no... by AlotOfReading on Tuesday November 14, 2023 @02:07PM (#38267939)
Let's use two of the examples I gave above. How would you go about modifying the silicon to support new features in the boot rom, and how do you get around the pigeonhole principle when that silicon vendor doesn't ship enough bits of OTP memory to store multiple keys? M-x butterfly doesn't quite get there, but maybe there's another Emacs command I could use?
Re:Rivian software update bricks infotainment system, fix no... by whalesalad on Tuesday November 14, 2023 @02:10PM (#38267978)
You choose different silicon? You get a different vendor? You build your own boards and chips?

For a vehicle that costs $100k+ it shouldn't be hard to double or triple the budget for onboard compute considering it is vital to the operation of the entire vehicle.

Re:Rivian software update bricks infotainment system, fix no... by AlotOfReading on Tuesday November 14, 2023 @02:27PM (#38268272)
Note that I never claimed the automotive world has functional people processes. Functional people processes make a lot of technical issues much easier, but they're usually off the table in traditional manufacturing. The security team insists on x requirements for security reasons. Hardware team insists on this chip because it's the only one that makes the budget work.

A shockingly large part of my job is telling both that one team won't be getting what they want and to work it out among themselves. Rinse and repeat between dozens of boards because the relevant teams don't talk to each other and none of them read the "design requirements to ensure we don't have to tell you no" doc either. One time they didn't even tell us there was a board until the end of December, when delivery was scheduled for Feb.

Re:Rivian software update bricks infotainment system, fix no... by whalesalad on Tuesday November 14, 2023 @03:54PM (#38269602)
I live in Detroit so what you’re saying does not surprise me in the least bit. I have some friends in automotive at some of the big 3, and suppliers, and I’ve heard some really terrible stories.
Re:Rivian software update bricks infotainment system, fix no... by mlyle on Tuesday November 14, 2023 @02:15PM (#38268051)
If the comment I replied to originally contained a mention of A/B partitions, I missed it.
Re:Rivian software update bricks infotainment system, fix no... by hef19898 on Tuesday November 14, 2023 @01:27PM (#38267360)
I am still not sure why I would update software on car, a piece of hardware that, IMHO, shoupd be able to run air gapped 24/7. Exceptions: recurring bugs, GPS maps and security updates. All of which can be done either during service (preferred, if they brick it, they are liable) or by plugging in something. OTA updates just seem completely pointless...

Edit: Also, why the heck isn't the entertainment system completely air gapped from the software running the car?

Re:Rivian software update bricks infotainment system, fix no... by enragedcacti on Tuesday November 14, 2023 @01:47PM (#38267657)
Rivian consistently ships a lot of new features and improvements, you can see the changelogs here [1]. I think you can pretty fairly critique a lot of them with: They are just solving a problem that they created by making it too techy, or they are shipping stuff they should have completed before releasing the product. I do think its hard to argue that the updates aren't adding anything of value though. There's convenience stuff like pet mode or bird's eye camera view that were added after release, but there are also things like new driving modes (soft sand and snow) or improved DC charging curves and smarter battery conditioning that legitimately improve the quality of the product as a vehicle.

> Edit: Also, why the heck isn't the entertainment system completely air gapped from the software running the car?

As for this, the entertainment system can control basically every feature of the car and is often the primary or only way to accomplish certain things. Even in much much dumber cars the infotainment is still part of the CAN bus and is able to interact with the rest of the vehicle.

https://rivian.software/category/public/

Re:Rivian software update bricks infotainment system, fix no... by hef19898 on Tuesday November 14, 2023 @02:18PM (#38268122)
Funny, our 2020 MY Jaguar controls car functions from the digital screen in front of the driver, the middle console screens only control AC, entertainment, phone, navigation and other non-car related stuff. No idea how the architecture looks behind all that so. But seriously, even if on the same bus, just don't the media player, radio and connected phone access to the systems actually running the car from engine to brakes. And please, please, finish developing the embedded software running on car before shipping said car. Then it can be air gapped, if not it requires OTA and internet access, raising all kinds of security issues...
Re:Rivian software update bricks infotainment system, fix no... by CamperBob2 on Tuesday November 14, 2023 @04:56PM (#38270397)
And please, please, finish developing the embedded software running on car before shipping said car.

Sorry, but things no longer work that way, and never will again. This is a good thing, as long as processes are improved to avoid situations like this one.

Re:Rivian software update bricks infotainment system, fix no... by lotsoweiners on Tuesday November 14, 2023 @07:14PM (#38271684)
> This is a good thing, as long as processes are improved to avoid situations like this one.

How do you figure. I can’t think of a single thing in my vehicle that could be improved by software. When I buy a car I’d prefer it was done.

Re:Rivian software update bricks infotainment system, fix no... by LoganDark on Tuesday November 14, 2023 @11:32PM (#38273383)
It's supposedly a good thing because more features can be added over time, but if they were features worth adding in the first place, the car should have shipped with them already.
Re:Rivian software update bricks infotainment system, fix no... by hef19898 on Wednesday November 15, 2023 @03:16AM (#38274441)
That is the big difference between hardware and software engineering: Once hardware is shipped, there is nothing you can do besides repaurs and retrofits requiring a workshop. Software can updated and changed today anytime.

My biggest issue with modern cars, and it seems this is spreading to other embedded systems than cars, is treating those as software: connect them to the web and run OTA updates everytime you need to fix a bug. That requires some form of inzernet cinnection, which requires regular security updates, which require OTA update capability. All that because, bluntly, software devs cannot be bothered with just finishing software running on non-connected hardware that just works, not can they be bothered following hardware development in case hardware is the main component, as in cars. And no, no car manufacturer is a software company, nor phone maker like Apple.

The proper way of fixing software bigs in automotive used to be, again thank you Tesla for breaking something that just worked, to recall the affected cars to a workshop to conduct the software update. Honestly, the only thing on a car that should be updated by the owner is maps for the GPS unit.

Re:Rivian software update bricks infotainment system, fix no... by jacquesm on Wednesday November 15, 2023 @01:16AM (#38273840)
Things work like that just fine: buy an old car and call it a day.

The less software in my car the better.

Re:Rivian software update bricks infotainment system, fix no... by jacquesm on Wednesday November 15, 2023 @01:14AM (#38273830)
You may not have a say in the matter. Most cars are on some kind of IoT private network with their own cell modems and if the manufacturer decides to push an update there isn't much you can do about it short of ripping out the cell modem. Which may well have unpredictable consequences.
Re:Rivian software update bricks infotainment system, fix no... by MichaelZuo on Tuesday November 14, 2023 @12:50PM (#38266779)
The 'early release tests' weren't testing an identical copy of the actual update?
Re:Rivian software update bricks infotainment system, fix no... by DannyBee on Tuesday November 14, 2023 @05:04PM (#38270479)
It's probably closer to:

The test vehicles accept test/prod signed versions

Regular vehicles only accept prod signed versions

They are otherwise identical.

The test vehicles were sent test signed versions

The prod vehicles were sent the exact same update, signed with test.

This would not be uncommon since the test vehicles probably occasionally run test releases for debugging.

Further, the update is probably multiple signed pieces, and the only part accidentally signed with test was likely infotainment software.

Or something like this.

It's hard to believe they wouldn't test sending badly signed updates, so i have to imagine it's a particularly weird badly signed update.

In other words, i would not assume they are idiots.

Re:Rivian software update bricks infotainment system, fix no... by AlotOfReading on Tuesday November 14, 2023 @05:56PM (#38271025)
It'd be pretty silly to implement an OTA scheme that didn't check signatures before installing updates. That would mean any random attacker could soft-brick the module by sending an invalid image, which a development image should be to a production vehicle.

You could get this situation if the application code accepted signatures the bootloader does not though. I can imagine that accidentally occurring.

Re:Rivian software update bricks infotainment system, fix no... by intern4tional on Tuesday November 14, 2023 @07:43PM (#38271917)
Likely not their code.

OTA is generally developed by tier 1, so this is probably a bug in the tier 1's code. (Samsung, Panasonic, Sony, etc are common tier 1s in this space.)

Re:Rivian software update bricks infotainment system, fix no... by DannyBee on Tuesday November 14, 2023 @09:46PM (#38272726)
Sure, but it could also be signed binaries inside the OTA, a variant of what you are talking about.

IE OTA is a signed package, inside package are also signed binaries. OTA itself is properly signed, a single binary (infotainment) is signed with wrong key.

While most OTA verifiers will verify the OTA signature (which this would pass), most don't verify the individual inside-package binary signatures at install time, only at runtime.

Re:Rivian software update bricks infotainment system, fix no... by AlotOfReading on Wednesday November 15, 2023 @01:54AM (#38274021)
That's not how I've ever implemented OTA, but I'll grant that it's possible if it was designed by someone with no idea what they were doing. Certainly not a good OTA process though, for this among many other reasons.
Re:Rivian software update bricks infotainment system, fix no... by mlyle on Thursday November 16, 2023 @04:07PM (#38295552)
It makes sense for each component to check signatures of its code to prevent various kinds of attacks -- e.g. someone coming and reflashing just infotainment or motor controllers with something malicious.

So, OTA update comes in, containing a bundle of software for different subsystems. It's sent to different subsystems. Then, those subsystems check integrity at startup, but one subsystem's bootloader isn't happy because the firmware looks to be invalid.

Re:Rivian software update bricks infotainment system, fix no... by DannyBee on Thursday November 16, 2023 @09:33PM (#38298635)
Uh, this is a perfectly normal ota process when dealing with multiple embedded systems.

It seems silly to off the cuff assert that it is a bad process or only done by those who have no idea what they are doing.

Re:Rivian software update bricks infotainment system, fix no... by refulgentis on Tuesday November 14, 2023 @01:42PM (#38267587)
Rollouts don't solve problems, they limit who they effect.
Re:Rivian software update bricks infotainment system, fix no... by mlyle on Tuesday November 14, 2023 @02:14PM (#38268048)
Is not reducing the effective cost of a bad update by 10x or more worthwhile?

Sure, but if you are rolling out to 1% of users per hour, you detect the problem in a couple of hours and much fewer than 2% of users will have applied the update. This is a relatively small support problem.

While if you roll out to everyone at once, you'll detect the problem sooner (within an hour) but have 10x as many affected.