Automakers effectively pulled the rug out from under their chip suppliers by asking them to severely cut production, then turning around and asking for a lot more than their original quantity. In general, those suppliers canceled their supply capacity with their factory partner, so when automakers turned around and placed new orders, those factories already sold $xx months of that factory time to someone else, making it hard for the semiconductor manufacturers to get back to their own production capacity and fulfill the auto industry’s orders.

https://www.freep.com/story/money/cars/2021/06/15/car-chip-s...

You’re exactly right.

Somebody at a major chip company told me that they went looking for, and found, new customers after the automotive customers canceled their orders.

So not only did the car companies ordered more than before after they realized that demand was not dropping, even if demand for them had returned to what it used to be, the total demand from other customers is now higher too.

TAM for semiconductors turned out to be wayyyyyy bigger than the world's manufacturing capacity for semiconductors. I don't know how this caught manufacturers by surprise, but perhaps there's bottlenecks in talent, supporting industry and infrastructure that I'm not aware of that would have made additional expansion difficult previously.

There’s just not flexibility built in to the supply chain because flexibility adds cost.

When things go well this just in time manufacturing is lauded as a huge money saver, when they aren’t… you get now.

The business of running everything to the wire on credit and working “cost of capital” so much into business decisions gets us here.

Blame this on just-in-time supply chains where no one wants to hold inventory. It’s been MBA dogma for the last two decades.

Seems like we are dealing with the opposite issue here where factory production is booked many months in advance and can’t quickly adapt to new demand.

The factory is not in trouble, their customers are.

Neither JIT nor many other things factors in "global pandemic," but proper JIT that knows how to scale up/down theoretically could handle things, its just the human element that fucks up the system. Yeah beancounters are beancounters but trying to blame JIT is really off course.

I disagree, respectfully. Not that I think everyone should inventory a year of everything. But the ethos of JIT is that you can treat your supply chain as if it’s infinite, and this keep it close to your projections.

Automotive got hit especially hard since any feature that requires electrics has the microprocessor embedded in that feature. Very cost effective-you don’t ship extra cpu power for unused features.

Tesla did the opposite though—centralized processing and over allotted cpu cycles by also made it easy to upgrade via software only.

Isn't the problem that is is not easy to scale some things up and down? This sounds a lot like no true scotsman.

> This sounds a lot like no true scotsman.

The claim is more that they are bad planners, orthogonal from whether they use JIT or not.

They're not doing proper supply management. You can do proper supply management with JIT or without JIT. You can also do improper supply management with JIT or without JIT.

For it to be "no true scotsman" they would have to be claiming that "doing JIT" is a solution to supply management. And they're not claiming that.

I guess you are partly right, but parent said "proper JIT" would work well without any evidence to what that means. Surely not everyone is doing the "improper" JIT.

I remember reading somewhere that automakers trying to get the governments involved to have them to lean on the chip suppliers to give the automaker the highest priority over other clients.

Why isn’t the result of those actions that those auto manufacturers either pay more or go without, rather than every buyer paying more and/or going without?

Because all capacity is being used. So it's not that today's capacity costs more (this capacity was sold and paid for months/years ago), it's future capacity that costs more. Until there are more chip factories, this is going to be an ongoing problem for the chip industry.

> Automakers effectively pulled the rug out from under their chip suppliers by asking them to severely cut production

Automakers only make few percents of MCU market. Just the most profitable one. Plus, a lot of MCU makers have own fabs.

It's very unlikely they took "months" of factory output. They just got their small batches to wait until the tsunami wave of mainstream part runs is done.

Sounds like someone at the car manufacturing has a shit ton of IOT stocks and knew exactly what they were doing

No, they're just being short-sighted and revenue-first. Think about how the rental car companies sold their idle car stocks and now can't rent cars reliably to anyone.

That's what happened, but faced with insolvency, you gotta do what you gotta do. You might really need those cars, but if selling it is the only way you'll make rent and payroll for the month, you gotta sell the cars. It's not a matter of short vs long term thinking, it's a matter of bankruptcy!

The bigger the BOM, the higher the probability something is not available, blocking the product from being fully manufactured.

Cars have bigger BOMs than IoT devices.

Given the long lead times, tech companies are scrambling to use alternative chips: first finding something functionally equivalent and pin compatible; that failing, qualifying something close enough and redesigning PCBs with an updated component footprint; last case scenario, porting FW to a different chip and redesigning circuits to accommodate.

Traditional car companies largely rely on suppliers to redesign electronics (at even greater cost and lead time). There's also more liability with cars so last minute redesigns have to be taken very seriously.

> Traditional car companies largely rely on suppliers to redesign electronics (at even greater cost and lead time).

Just to put this in perspective for some readers, I used to work at one of the largest automakers and suppliers would bend over backwards for us. They would shut down the line and retool for us when we needed something quickly. But even when things were running smoothly, some parts had a 12 month lead time and there was nothing that could be done to expedite that.

I remember one part that we shared with many other automakers had a recall. It took the industry over 3 years to supply all of those parts to consumers. Not airbags, this was almost 20 years ago.

All of that is without a design change. Now add a design change on top and it can easily move a 3 month lead time to 12 months or more.

I do consulting for another automaker now and one of our chips has become unavailable. Actually, at the beginning of the pandemic the price jumped almost 10,000%. We had no choice to pay it, but that also killed all of the margin on that particular part so we immediately started redesigning the PCB for a new chip. The software had to be rewritten for the new chip; several case components had to be redesigned to accommodate the new PCB, testing, mass production, etc. That was over a year ago and the part is just starting to roll out now.

So, I think PlayDate's messaging is completely reasonable.

I suspect that IoT devices also have a much shorter turnaround when they need to redesign the board to use the chips available.

Anecdotally I think this is right on the money. I've been buying lots of LoRA hardware over the course of the year and I think I'm up to 9 different permutations of the "same" board via swaps of the LoRA SXnnnn/ESPnnnn/GPS Mn units for whatever is currently available. Turns out to be a good way of discovering firmware bugs.

I'm guessing BOM doesn't mean Byte Order Marking here?

In this case, Bill Of Materials, as in a list of all chips, resistors, capacitors and other electronic components.

Bill of Materials - all the components required to build a circuit

Bill Of Materials

All the big industries got scared when COVID hit thinking consumer spending was going to plummet so they drastically cut orders.

Meanwhile the IoT space has been growing rapidly in the past few years, and those suppliers didn't cut their orders. IoT devices like the ESP8266 can sit in a warehouse for the next few years before being sold for use in some smart lightbulb, but automotives will lose their value as they become last year's model.

With the supply chain disruptions and increasing consumer spending automotive industries basically lost their reserved spots are struggling to claw back their manufacturing and shipping capacity.

So basically in this game of musical chairs the automotive industries stood up while the IoT space stayed sitting, and then the supply chain disruption removed the chair the automotive industries had been sitting in before...

This doesn’t serm right. The car manufacturers would surely just buy their way into the front of the line, if it was just this. Chips are a fraction of a car’s value and the cost of stopping car production lines must be a lot higher than buying slots in s chip production facility.

"Buying your way to the front of the line" may work if it involves paying $18 for a part that normally costs $15. (Even so, the automakers will scream bloody murder over much less.)

That's not what's happening. What's happening is that the $15 part now costs $150, and is only available from sketchy Asian brokers that aren't on anyone's list of approved vendors.

The semiconductor houses had better unfuck themselves SOON, or 2022 will go down in history next to 1929. I don't think anyone understands how serious this situation is getting.

Between flooding and Covid-19 lockdowns, the semiconductor houses aren't, like, choosing to be shut, they're closed because of global climate change. This is only going to get worse as the climate changes dramatically within our lifetimes.

> The car manufacturers would surely just buy their way into the front of the line, if it was just this.

Those older nodes "chairs" that the automotive industry used are less profitable in general. One doesn't simply spin down and spin up a semiconductor fabrication plant.

>> the supply chain disruption removed the chair the automotive industries had been sitting in before...

Replacing the (old) chair is easily in the billions of dollars. Not everyone can foot that kind of bill.

In that case it was an incredibly stupid move for car manufacturers to wind down orders for semiconductors, even in the face of covid. The down side risk is huge.

A charitable interpretation of that decision would account for the fact that they have more information than you do about the situation.

Well, they may have had more information, but it certainly wasn't good information.

> The car manufacturers would surely just buy their way into the front of the line, if it was just this. Chips are a fraction of a car’s value and the cost of stopping car production lines must be a lot higher than buying slots in s chip production facility.

There is no excess capacity at fabs, so if someone buys their way to the front of the line, the fab is going to miss delivery dates on all the contracts it currently has.

Fabs do have that option, but it's going to cost an enormous amount. Automakers have to pay enough that the fab makes enough money to compensate for the loss of reputation (it won't be a secret that they let someone skip the line), even after subtracting any contractual fees brought about by missing delivery dates on other contracts. That's why you aren't paying slightly more, you're paying orders of magnitude more. I would also be unsurprised if everyone who can get in an order is paying a little extra to raise the contractual fees for missing a delivery date.

A single jumped order can make 100 other contracts deliver late. That's a lot of fees.

IoT are OK with high risk and move very fast with startup mentality. You have qty zero of CPU model XYZ in stock and qty essentially-infinite of CPU model ABC in stock? Our new product, which will have zero support after shipping, will ship with CPU ABC which is in stock.

Car mfgrs are in the risk adverse capital preservation mode and it takes years for changes to be made. You only have model ABC CPU in stock? Has that undergone enough approval and testing processes to be an automobile ABS controller? No? Then we wait. It'll take years for model ABC to be approved -or- similar years for us to design a new brake controller system.

If you ship an IoT product its assumed it'll have zero aftermarket support, be wide open to new security holes, and probably stop working in a couple years at most. Cars are expected to have parts available for 20 years, never be hacked, and reliably operate in a life critical environment for decades.

If the IoT temperature sensor is 2 degrees high, nobody cares anyway, ship it and fix the software. If the automotive brake controller only applies the brakes 99.9% of the time perhaps because it has less memory so the stack crashes under weird and rare conditions, it'll pass trivial levels of testing while killing lots of people causing a financial and PR nightmare...

A better comparison to automotive would be aviation, or perhaps ocean shipping.

Maybe IoT garbage can ship with just about any chip that will run Linux, while cars need specifically certified components?

Not only certified (e.g. ASIL*) but long term availability.

* https://en.wikipedia.org/wiki/Automotive_Safety_Integrity_Le...

Which neatly explains why STM32 microcontrollers are in such high demand -- most of their microcontroller portfolio has a longevity commitment of at least 10 years.

https://www.st.com/content/st_com/en/about/quality-and-relia...

Espressif also commits to 8 years exactly because they want to capture lazy manufacturers.

>"Espressif also commits to 8 years exactly because they want to capture lazy manufacturers."

It's not about laziness, it's about certainty, certifications, and saving money on unnecessary redevelopment and testing. If each of the ICs on your board gets discontinued every two years, and you have 20 ICs on your board, it means that you will have to go through redesign and re-testing as often as you buy components, even if you're doing ten batches a year.

I don't think so because there's rarely a drop in replacement unless you designed your board ahead of time for multiple options (of course exceptions exist, there's drop in replacements for many devices including the esp8266 I mention below).

But for example, an ESP8266 hasn't really lost much value in the past few years, and is unlikely to lose significant value if it sat on a shelf for another few years, so they don't mind a bit of an overstock.

Meanwhile the current year car model will lose a ton of it's value when the next model comes out so they don't want to risk over-stocking it.

If you've got a low-chip-count product and an agile business structure, when Chip A is unavailable you can respond by redesigning your PCB to use Chip B.

But if you've got loads of chips on different boards from different suppliers, a structure suitable for a large business, lengthy supply chains, complex certification requirements, and you can't afford to pay much more? Well, that's going to make things quite a bit more difficult :)

This is partially true. Vehicles are needed but most of the computing the modern ones do is completely unnecessary for the primary function and purpose of the vehicle. There is really nothing important I can do with my Tesla that I can’t with my 1980s Ford ranger. There are definitely important things I can do with the old pickup truck that I can’t do in the Tesla. Until we reach a point where the car can legally drive you home from the bar while you are passed out in the back most of the stuff chips are used for in vehicles pretty much is for sales purposes and turns it into a giant smartphone. Add in the whole remote exploit thing about some of the models of different 2000s models with dash computing that never get firmware updates but still have remote network connectivity and it’s actually a pretty shitty idea overall until actual self driving cars are a thing. Before anyone mentions that you can track your car if it’s stolen, it’s important to remember most of the thieves going after new cars will back up to the vehicle with a metal box truck lined with some kind of faraday material then pull your car into it.

Having an electric vehicle doesn’t mean it needs the fastest hardware. You can run the entire base mechanics sensors and outputs of an electric vehicle on an Arduino with some extra breadboards.

> pretty much is for sales purposes

Features like blind spot detection, backup cameras, and adaptive cruise control are pretty valuable for general collective safety reasons.

It's not even legal to sell a new car in the US without a backup camera.

The automotive industry presumed they were higher delivery priority than they really were. By volume they were not thus they had lower priority when things got squeezed.

The auto industry also had unreasonable chip reliability demands - physics doesn't care that they wanted 30-year life while also having 7-10 nm CPU performance - you can NOT HAVE BOTH!

IoT is primarily NOT 7-10 nm performance nodes - AVRs and many ARMs are not at that level - you can get by with larger geometries which can be made on older processes. So not so much shortage issues.

No, there are legitimate shortage issues, and the "Automakers screwed up their forecasts" excuse is starting to wear awfully thin.

Intel FPGAs aren't typically used in cars, for example, yet we're seeing quotes of $500+ on parts that used to cost $50. Lead times? "Don't ask."

If there's a shortage and buyers willing to pay a premium in one segment, it affects all other segments because at least part of manufacturing capacity is interchangeable (with switching costs, but those are bearable if someone wants to pay) and it becomes profitable to randomly shut down production of various niche parts to save some car manufacturers arse, or have essentially a bidding war between those niche parts or car parts in order to discard any customers which aren't willing to pay increased prices.

Automakers more or less started the shortage, as a result of that everyone is in squirrel mode buying up what they can because they rightly fear having to reengineer and revalidate every single production run. That our new economy has a greater thirst for chips just compounds the issue.

FPGAs are likely short because they are a substitute for SoCs in expensive products.

Everything is sloshing around.

If you want to deliver that part to beat your competitor this year, you will find a substitute.

Automakers are notoriously looking for years-old chips because they don't update their products the same way. I believe that stock is just dead, more or less.

Dirty little secret: none of the leading-edge sub-10nm parts are of any use whatsoever without peripheral, power management, and analog parts built on those "dead" processes.

The Decoder podcast had on the CEO of Anker and he explained (IIRC) that chips coming off of 12" wafers are impossible to get, while chips from 7" wafers are plentiful. The problem is recertification, which for autos can be a very long process.

Why that occurred is another question.

This is just the quoted lead time. When the cycle eventually turns, there will be an enormous wave of push outs and cancellations and those who actually wanted the full order will get the product much sooner

Nope, they're having trouble too. I was on the phone with an embedded cellular modem module manufacturer earlier this year. The part we spec'd was unavailable because the underlying modem device was on allocation for so long that the module manufacturer decided to just discontinue the device rather than say it would be out of stock for two years.

I don't understand why car manufacturers haven't just started remanufacturing some of their older (say 2010) model cars that don't rely on so many computer chips. I'm sure there's lots of costs to retooling their factories, and these less computerized cars are going to be less desirable to most customers, but with car prices currently 150% of what they "should" be, it seems like it would be profitable for at least one company to go the "low chip" route.

My 2011 car has ECUs for: engine, brakes, ABS, air bag, air conditioning, alarm. All of those are connected to a gateway, that is another ECU. Do you want parking sensors? That's another ECU. Multimedia? That's one or two. Cars of 2000 have not much less ECUs. Consider also that components to build such ECUs are now totally obsolete and out of production.

My old 1997 Town Car has about as many controllers as my 2011 Crown Victoria does (many of them are even the same parts), and the 2021 300S I just ordered doesnt have many more.

Though the 1997 Town Car, had many more controllers than a 1997 Taurus for comparison.

It takes close to a year to re-tool a factory for a different auto model. The suggestion to just produce an older model just doesn't make sense in time or money.

Some manufacturers already did remove advanced vehicle features.

https://www.bloomberg.com/news/articles/2021-05-06/chip-shor...

BMW is pulling the touchscreens off of certain models:

https://www.autoblog.com/2021/11/05/bmws-losing-touchscreen-...

Slightly confusing since it's not the removal of the entire infotainment system, just the capacitive controller. The system is still controllable with the joydial. Seems like that's not really a $500 cost center, more of a way to appease the customer.

And some makers like Mazda pulled the captouch off their IVI with the recent generation of cars pre-COVID, it just wasn't worth the cost.

For Mazda it was also at least theoretically to reduce driver distraction and increase safety. I mostly believe them on that point, given how much they focus on safety ratings in other ways.

Some of them were also mounted pretty far from the driver so they weren't easy to reach, and they had a dial to alternatively interact with them. The first few times I drove one, I didn't even know it was a touch screen.

On the earlier Mazdas the touchscreen actually disabled itself when in drive. Dial only.

Seems like most people didn't need it anyway.

My guess would be they won't be regulatory compliant. Few cars have already removed non essential things like touch screen.