eMMC flash is typically a ball-grid array part and it's soldered down hard to the motherboard of the processor. Almost every mobile device has the same setup and it's tricky and expensive to rework this chip.
You need to carefully desolder it off the board, reball the part with the correct masking stencil, then reapply it without torching the rest of the board, missing a solder connection, or zapping any other component with static electricity.
You can see a similar process done on an iPhone here: https://www.youtube.com/watch?v=y4M9uAZlbK4
You can understand why Tesla would rather replace/refurbish the board than try to field-repair it.
Sure refurbish the processor module and replace the eMMC at a proper rework station but replacing the whole media unit esp when the failed part is on a removable board just seems like a huge waste to me esp when the customer is footing the bill.
In my experience, removable SD wouldn’t hold up to an application like this.
If it were held in and held against some robust contact pins it likely would be fine, even under automotive conditions. There are SIM card slots that are used on heavy vehicle computers and controllers (the ones that run the engine and so on) that are robust enough to hold a SIM card in place for years of excessive daily vibration.
I suspect that designing reliable spring contacts isn't as easy as it seems...
The trade-offs for a high vibration environment are very different.
You are not wrong that a soldered eMMC (compared to a eMMC held by a ZIF or a removable SD Card) would be more mechanically stable and potentially faster (A fast uhs-ii sd card compared to a cheap slower eMMC for example), just saying that Tesla already use removable SD cards in production cars today (which doesn't brick your media unit if it dies or dislodges).
Storing the OS on a removable SD Card is (imo) a bad idea, but if they are really interested in these logs they already have a easier to replace removable storage "drive" they could write the logs to instead of burning up the eMMC.
Replace the board, then you can refurbish it in a specialized lab
Same reason why genius bars won't replace components, just boards
On the newer versions of the Media Unit instead of disabling writing the syslog to the eMMC they added more storage to the processor boards which isn't going to solve the problem just push it from appearing to further down the road.
To say that the choice to use eMMC is the fault is to say that ANY use of flash storage is a faulty decision.
If there is sufficient space on the drive to mitigate write amplification, then the media will last a very long time indeed. Longer than the drivetrain, certainly.
The problem is that Tesla doesn't delete logs it has collected from the cars. All Tesla vehicles (as I understand it) upload their logs more or less continually. Currently, the firmware on the cars doesn't delete those logs once they've been uploaded, presumably so that they're available when it is time to service the vehicle.
The disk filling up due to bad log file hygiene, coupled with the fact that the storage is flash-based, together cause the issue.
A simple firmware update to the eMMC chip could make it use a wear levelling algorithm that really does spread writes evenly over the whole chip. Current implementations simply use a sector till it goes bad and then using a spare till that goes bad, etc. After 1% or so go bad, the whole device rejects writes entirely.
With a decent implementation (like ssd's mostly do), you will never be able to kill a card in a lifetime.
Do you know that they toss the old media units? Are you sure they don't swap the entire media unit to simplify service procedures, and have have the old units inspected and refurbished (i.e. eMMC module replaced) for future repairs?
That sounds way more efficient than training all of the shop workers to diagnose and swap out individual parts of the media unit.
most of the time me and the other firmware folks were chasing elon's whims about what to do with firmware. where i should have been fixing critical issues in the system i was pulled off to do shit like add farting unicorns
The linked reddit post is from a user who claims to have worked for Tesla in the past, and is (presumably) not using their "real" reddit account to post it.
I don't read it as an insult.
Gotcha. That makes sense. Slashdot was never my thing. Thanks for clarifying; I had read it as bafflingly unjustified aggression.
Not sure of the benefits of an SSD over an eMMC. Guessing the former will have a more sophisticated FTL and other things.
But there's still the problem of a friction fit connection in an environment with a lot of vibration and contaminants. Seems like these would be good for use cases where the part needs to be removable and also have high write cycles, like a dash cam or data logger that has no other interface to a PC.
Also the eMMC interface exposes quite a bit of the underlying hardware (see mmc-utils on Linux). With SD cards I think you're generally stuck with some kind of SMART interface.
Am I the only one thinking about the recycling and cost implications of doing this?
I'm thinking they would have all the labor of taking it out, and all the labor of putting it in, but then much additional labor and repair time incurred by doing the component level repair instead of just grabbing another radio off the shelf. Recycling would be possible on the unit later and it could be installed as a refurb unit in another vehicle.
This has been a known issue for some time, and it's amazing Tesla hasn't fixed it yet. As explained in the video, the issue is simply the base linux OS has full syslog's still enabled, which over time burn own the eMMC chip used for storage. Currently Tesla just replaces the entire media unit when this happens, which is extremely expensive for them. In reality they only need to replace a tiny part instead of tossing entire boards.
The GOOD news is this issue is almost fully solvable through a software update. They just need to disable base syslogs, or at least only store logs in RAM while car is running (if they need those logs for debugging purposes, which I doubt). Long term, they should replace this eMMC with a proper SSD, or removable SD card that's easily serviced when it goes bad (as is the case in other parts of the car).
I think this must be an issue of visibility in the company, and the right engineer just isn't aware of this issue yet. I hope this gets more visibility and gets patched in a future software update.