News for hackers. Stuff that matters.
Re:A Single Component Can Brick Older Teslas by BluSyn on Tuesday May 14, 2019 @02:31PM (#19912416)
This issue is best explained on Rich Rebuilds' recent video here: https://youtube.com/watch?v=o-7b1waoj9Q&t=525

This has been a known issue for some time, and it's amazing Tesla hasn't fixed it yet. As explained in the video, the issue is simply the base linux OS has full syslog's still enabled, which over time burn own the eMMC chip used for storage. Currently Tesla just replaces the entire media unit when this happens, which is extremely expensive for them. In reality they only need to replace a tiny part instead of tossing entire boards.

The GOOD news is this issue is almost fully solvable through a software update. They just need to disable base syslogs, or at least only store logs in RAM while car is running (if they need those logs for debugging purposes, which I doubt). Long term, they should replace this eMMC with a proper SSD, or removable SD card that's easily serviced when it goes bad (as is the case in other parts of the car).

I think this must be an issue of visibility in the company, and the right engineer just isn't aware of this issue yet. I hope this gets more visibility and gets patched in a future software update.

Re:A Single Component Can Brick Older Teslas by joezydeco on Tuesday May 14, 2019 @04:58PM (#19913842)
"Replacing a tiny part" is trivializing the work necessary to do this, in my opinion.

eMMC flash is typically a ball-grid array part and it's soldered down hard to the motherboard of the processor. Almost every mobile device has the same setup and it's tricky and expensive to rework this chip.

You need to carefully desolder it off the board, reball the part with the correct masking stencil, then reapply it without torching the rest of the board, missing a solder connection, or zapping any other component with static electricity.

You can see a similar process done on an iPhone here: https://www.youtube.com/watch?v=y4M9uAZlbK4

You can understand why Tesla would rather replace/refurbish the board than try to field-repair it.

Re:A Single Component Can Brick Older Teslas by Crosseye_Jack on Tuesday May 14, 2019 @05:26PM (#19914047)
They have been replacing the whole media unit including the screen (and charging customers $2k-5k for the priv) where they could replace the processor module board (which is "just" a add-in board inside the media unit) keeping the rest of the media unit and lowering costs.

Sure refurbish the processor module and replace the eMMC at a proper rework station but replacing the whole media unit esp when the failed part is on a removable board just seems like a huge waste to me esp when the customer is footing the bill.

Re:A Single Component Can Brick Older Teslas by arcticbull on Tuesday May 14, 2019 @05:44PM (#19914161)
It feels like they should just turn off syslog in production builds, release it in a firmware update and offer a recall for everyone affected by broken board.
Re:A Single Component Can Brick Older Teslas by solarkraft on Tuesday May 14, 2019 @05:26PM (#19914044)
Luckily Tesla isn't very space or money limited with this pretty ciritcal component. An SD card would make life easier for everyone, as the logs could be easily read out and the booting OS changed - yay for actually owning the device you paid for. But wait: That's not something Tesla wants.
Re:A Single Component Can Brick Older Teslas by joezydeco on Tuesday May 14, 2019 @06:19PM (#19914381)
eMMC is literally the equivalent of an SD card soldered down to the PCB. It’s more mechanically stable and also way faster.

In my experience, removable SD wouldn’t hold up to an application like this.

Re:A Single Component Can Brick Older Teslas by naikrovek on Tuesday May 14, 2019 @07:38PM (#19914929)
An SD card might work fine if it is mounted properly. The slot found on a Raspberry Pi would not at all be sufficient.

If it were held in and held against some robust contact pins it likely would be fine, even under automotive conditions. There are SIM card slots that are used on heavy vehicle computers and controllers (the ones that run the engine and so on) that are robust enough to hold a SIM card in place for years of excessive daily vibration.

Re:A Single Component Can Brick Older Teslas by londons_explore on Wednesday May 15, 2019 @04:04AM (#19917288)
Whenever I drop my phone I see the "SimProcessor" service spinning up...

I suspect that designing reliable spring contacts isn't as easy as it seems...

Re:A Single Component Can Brick Older Teslas by naikrovek on Wednesday May 15, 2019 @09:23AM (#19919122)
Yeah, phones aren't designed to have robust SIM slots, they're designed to be thin, so they use very low profile SIM cradles that are not robust to impact.

The trade-offs for a high vibration environment are very different.

Re:A Single Component Can Brick Older Teslas by Crosseye_Jack on Tuesday May 14, 2019 @06:48PM (#19914584)
The actual logs Tesla seem interested in (including the stills of things they are interested in) get written to a removable SD Card before getting uploaded to the Tesla mother-ship.

You are not wrong that a soldered eMMC (compared to a eMMC held by a ZIF or a removable SD Card) would be more mechanically stable and potentially faster (A fast uhs-ii sd card compared to a cheap slower eMMC for example), just saying that Tesla already use removable SD cards in production cars today (which doesn't brick your media unit if it dies or dislodges).

Storing the OS on a removable SD Card is (imo) a bad idea, but if they are really interested in these logs they already have a easier to replace removable storage "drive" they could write the logs to instead of burning up the eMMC.

Re:A Single Component Can Brick Older Teslas by m-p-3 on Wednesday May 15, 2019 @12:06AM (#19916326)
Or just add an SD card reader on the board and output the logs there if they need it so much, and keep the rest on the EMMC.
Re:A Single Component Can Brick Older Teslas by raverbashing on Tuesday May 14, 2019 @05:14PM (#19913959)
This is the right answer

Replace the board, then you can refurbish it in a specialized lab

Same reason why genius bars won't replace components, just boards

Re:A Single Component Can Brick Older Teslas by pbreit on Tuesday May 14, 2019 @06:23PM (#19914409)
So it's not fixable with software update?
Re:A Single Component Can Brick Older Teslas by Crosseye_Jack on Tuesday May 14, 2019 @06:41PM (#19914519)
The under lying issue can not be fixed with software. the eMMC is simply running out of write cycles. BUT they can mitigate the issue by not writing the syslog to the eMMC allowing the remaining write cycles to last much longer (This change could be done with a software update).

On the newer versions of the Media Unit instead of disabling writing the syslog to the eMMC they added more storage to the processor boards which isn't going to solve the problem just push it from appearing to further down the road.

Re:A Single Component Can Brick Older Teslas by naikrovek on Tuesday May 14, 2019 @07:34PM (#19914899)
The problem is that there is significant write amplification as the used space on the partition approaches the total space of the partition.

To say that the choice to use eMMC is the fault is to say that ANY use of flash storage is a faulty decision.

If there is sufficient space on the drive to mitigate write amplification, then the media will last a very long time indeed. Longer than the drivetrain, certainly.

The problem is that Tesla doesn't delete logs it has collected from the cars. All Tesla vehicles (as I understand it) upload their logs more or less continually. Currently, the firmware on the cars doesn't delete those logs once they've been uploaded, presumably so that they're available when it is time to service the vehicle.

The disk filling up due to bad log file hygiene, coupled with the fact that the storage is flash-based, together cause the issue.

Re:A Single Component Can Brick Older Teslas by londons_explore on Wednesday May 15, 2019 @04:01AM (#19917275)
Most eMMC cards use really primitive wear levelling algorithms.

A simple firmware update to the eMMC chip could make it use a wear levelling algorithm that really does spread writes evenly over the whole chip. Current implementations simply use a sector till it goes bad and then using a spare till that goes bad, etc. After 1% or so go bad, the whole device rejects writes entirely.

With a decent implementation (like ssd's mostly do), you will never be able to kill a card in a lifetime.

Re:A Single Component Can Brick Older Teslas by semi-extrinsic on Tuesday May 14, 2019 @02:40PM (#19912495)
Wow, this is exactly the tutorials that you get on "How to get your Rasberry Pi to run stable for extended periods of time". Exact same failure mode, exact same fix proposed.
Re:A Single Component Can Brick Older Teslas by rush340 on Tuesday May 14, 2019 @04:14PM (#19913433)
> Currently Tesla just replaces the entire media unit when this happens, which is extremely expensive for them. In reality they only need to replace a tiny part instead of tossing entire boards.

Do you know that they toss the old media units? Are you sure they don't swap the entire media unit to simplify service procedures, and have have the old units inspected and refurbished (i.e. eMMC module replaced) for future repairs?

That sounds way more efficient than training all of the shop workers to diagnose and swap out individual parts of the media unit.

Re:A Single Component Can Brick Older Teslas by icelancer on Tuesday May 14, 2019 @03:21PM (#19912943)
An SD card has a much different set of problems, including physical contact issues. eMMC is an acceptable tool for the job, it's just complete negligence to not use a batch logging technique if you even need full syslog data.
Re:A Single Component Can Brick Older Teslas by rootusrootus on Tuesday May 14, 2019 @03:22PM (#19912959)
They use an SD card for the car logging, according to the Rich Rebuilds video, so it's doable. It just needs to not brick the car when it fails.
Re:A Single Component Can Brick Older Teslas by icelancer on Tuesday May 14, 2019 @03:24PM (#19912978)
That they use it in other applications doesn't make it automatically the best idea. SD cards are fairly unreliable compared to the other options, and as Rich's friend points out, smartphone manufacturers use eMMC and don't have issues because they understand wear cycles.
Re:A Single Component Can Brick Older Teslas by pfortuny on Tuesday May 14, 2019 @02:36PM (#19912460)
It is a bit baffling that a company which enables “autopilot” OTA does not do this as well...
Re:A Single Component Can Brick Older Teslas by nm2259 on Tuesday May 14, 2019 @03:31PM (#19913030)
If this anonymous coward's characterization of Tesla is true and accurate, it's not surprising https://www.reddit.com/r/EnoughMuskSpam/comments/99sbwa/form...
Re:A Single Component Can Brick Older Teslas by village-idiot on Tuesday May 14, 2019 @04:52PM (#19913789)
The wording makes it sound like you're calling the parent comment an anonymous coward. I also suspect a lot of people didn't catch the Slashdot reference.
Re:A Single Component Can Brick Older Teslas by JaimeThompson on Tuesday May 14, 2019 @05:45PM (#19914170)
Maybe he was hit / smashed over the head with an OPEN SOURCE CD?
Re:A Single Component Can Brick Older Teslas by PhantomGremlin on Tuesday May 14, 2019 @11:51PM (#19916258)
Wow. Thanks for that link. Some amazing inside stuff. My favorite anecdote:

most of the time me and the other firmware folks were chasing elon's whims about what to do with firmware. where i should have been fixing critical issues in the system i was pulled off to do shit like add farting unicorns

Re:A Single Component Can Brick Older Teslas by arkades on Tuesday May 14, 2019 @04:21PM (#19913497)
Why did your choice of words include “anonymous coward”?
Re:A Single Component Can Brick Older Teslas by function_seven on Tuesday May 14, 2019 @04:30PM (#19913590)
On Slashdot, "anonymous coward" was the default username for anyone posting without being logged in.

The linked reddit post is from a user who claims to have worked for Tesla in the past, and is (presumably) not using their "real" reddit account to post it.

I don't read it as an insult.

Re:A Single Component Can Brick Older Teslas by arkades on Tuesday May 14, 2019 @05:13PM (#19913956)
> On Slashdot, "anonymous coward" was the default username for anyone posting without being logged in.

Gotcha. That makes sense. Slashdot was never my thing. Thanks for clarifying; I had read it as bafflingly unjustified aggression.

Re:A Single Component Can Brick Older Teslas by castratikron on Tuesday May 14, 2019 @02:40PM (#19912503)
Problems with an SD card is that it will wear out even faster than an eMMC due to lack of FTL, and also friction fit instead of soldered connections means the part is susceptible to wiggling out of the socket and also corrosion between the contactors.

Not sure of the benefits of an SSD over an eMMC. Guessing the former will have a more sophisticated FTL and other things.

Re:A Single Component Can Brick Older Teslas by JaimeThompson on Tuesday May 14, 2019 @05:44PM (#19914163)
Multiple companies make high endurance SD cards for things like dash cams and automobiles. Take, for example, WD https://www.westerndigital.com/products/embedded-removable-f...
Re:A Single Component Can Brick Older Teslas by castratikron on Tuesday May 14, 2019 @06:57PM (#19914640)
Didn't know that, thanks.

But there's still the problem of a friction fit connection in an environment with a lot of vibration and contaminants. Seems like these would be good for use cases where the part needs to be removable and also have high write cycles, like a dash cam or data logger that has no other interface to a PC.

Also the eMMC interface exposes quite a bit of the underlying hardware (see mmc-utils on Linux). With SD cards I think you're generally stuck with some kind of SMART interface.

Re:A Single Component Can Brick Older Teslas by at-fates-hands on Tuesday May 14, 2019 @03:42PM (#19913143)
> Currently Tesla just replaces the entire media unit when this happens, which is extremely expensive for them. In reality they only need to replace a tiny part instead of tossing entire boards.

Am I the only one thinking about the recycling and cost implications of doing this?

Re:A Single Component Can Brick Older Teslas by silversconfused on Tuesday May 14, 2019 @06:19PM (#19914377)
Well, if you have some thoughts on it, I am curious what the other approach would look like. What are your thoughts?

I'm thinking they would have all the labor of taking it out, and all the labor of putting it in, but then much additional labor and repair time incurred by doing the component level repair instead of just grabbing another radio off the shelf. Recycling would be possible on the unit later and it could be installed as a refurb unit in another vehicle.