One of the parts was devastatingly expensive to substitute. It was a very low noise JFET which was being used as a high impedance amplifier buffer. We had to evaluate ten different parts from other manufacturers then hand select a suitable part with the right curve and noise figure parameters.
To get 10 usable parts meant we had to buy 10,000 parts in, buy $40k worth of kit (HP FTW) and spend a month doing it. The BOM cost went to $6000 a FET from $50. Fortunately being in defence sector they just burned the cash while we redesigned the entire front end of the device which took months of qualification and test.
Automating this test process actually started my software career.
Edit: also I’ve got to stab Six Sigma and other process methodologies in the face here. They pushed elimination of waste and metrics that drove that to the front of business success. This meant eventually relying entirely on just in time manufacturing and supply chains. Coming from a defence background this was a disaster from the old days of buying in stock for the entire lifetime of the devices in the field.
I self eliminated and went and worked for a web design startup who’s process involved being drunk and winging it and it was more effective.
This is relatable even to hobbyist electronics tinkerers, at least to those dabbling in low-noise measurements.
https://searcherp.techtarget.com/definition/bullwhip-effect
While a lot of people are saying the main issue is the "lean manufacturing" or "just-in-time manufacturing" supply chains, they do not understand manufacturing supply chains. These types of solutions (JIT or Lean) are designed to prevent the bullwhip effect and ensure that suppliers have predictable and constant demand, rather than feast or famine.
Without just-in-time manufacturing, there is significant waste and it is even harder for suppliers to predict what they should be making.
This current bullwhip is going to be horrible on suppliers after the supply catches up with demand and then the demand collapses as so many have bought years in advance.
Fabs do not work on jit/lean order principle - if you up your order volume, you can't simply scale it up on demand the next month - even if your design is done and taped out. Fabs aren't sitting around idly waiting for orders nor are they overprovisioned "just in case".
An example of this would be AMD and TSMC. AMD books capacity 3 years in advance for which TSMC designs the entire fabrication facility around, orders the lithography machines (which have a 12 month lead time and can't scale up on a dime) and assembles and staffs those same facilities to produce the wafers. It's not a general-purpose fab as most think. There are rumors that Apple booked m1 fab capacity (on TSMC 5nm) in 2018. For the chip inside a car for example, it is built on much older process, and when 5 of your biggest customers "JIT" reduce their order volume for the next 3 months you simply put those staff elsewhere (probably on the 6x higher margin cutting edge fab you just got booked and prepaid for) since anyone else who might want that process and capacity has already has their orders being produced.
Due to the sheer capital requirements (and consequent commitments required) and multi-year forecasting involved in simply getting what you think you'll need, let alone having extra, it's not hard to see why a small increase in global demand would cause a dramatic shortage. JIT works in Automotive supply chain where you can request that your bearings get made 10 days before they're delivered to the factory, but not when there is an 18 month latency on "scale up" request/response.
It's applying JIT principles to inherently non-JIT supply chains.
Now, the whole world is trying to get TSMC to invest in other locales in advance of Chinese invasion of Taiwan at the same time that NATO is funneling arms and personnel to Ukraine ahead of Russian invasion. Work isn't getting done on expected timelines because of unprecedented disruption caused by responses to covid.
Meanwhile, down at the bottom of the pile, when airlines cancel my flights for unprecedented problems, I wait 3 hours on hold because they don't have enough people working support.
All this to say that, yes, the bullwhip is going to be horrible, but so is not being able to perform your function in the economy because your required inputs are unobtainable. There's no graceful path for a system with so much volatility. Hopefully we can all stomach the creative destruction.
If that were true, they would have ensured the chip manufacturers had constant demand, rather than a famine period followed by a huge feast.
The truth is that just in time is being used as a cost minimization procedure, by offloading the demand risks into the suppliers and increasing any feast or famine event. That maximizes profits as long as there isn't an emergency, what seems to be the favorite administration goal of the current age.
These chips are not fungible commodities. In the automotive space chips are selected based on a number of things including feature set - how many CAN interfaces does it have, how many GPIO, how many PWM, what kind of brown-out handling, what voltage range, thermal rating, etc... Software is then written where a portion of the code is specific to that chip. There are cases where an OEM get really good pricing on a bigger (overkill) chip and helps a supplier get those at good prices too, but usually they just beat you over the head for the lowest cost. Even if everyone adopted a few larger parts and used just those, they're still usually made by a single chip company.
This doesn't exist now for the same reasons it probably wouldn't work in the future: You can save significant amounts of money if you cut the pieces out of the "standard" that you don't actually need. Chips are cheap to make, in bulk. It's almost entirely a fixed cost/area. That's the magic. So, it's almost always cheaper to make a custom chip than to pay for some bloated standard containing large sections that will never see power, especially when chips are something like 40% of the BOM, for modern cars.
No knowledge of the automotive industry, but that sounds impossibly high.
I think there isn't a bullwhip effect on batteries, which is the main cost associated with TVs.
The 72 month loan for a $50,000, 2019 F-250 will be completely underwater, and the car owners may decide to literally send the cars “underwater” to get out of the loans.
Based on what happened in places like California with mortgages, they will default one way or another.
Used EVs used to be pretty dang affordable already (I got a Leaf and a Volt for under $10k each) before all used cars became more expensive and the price of gasoline increased, increasing the demand for EVs.
It’s just about scaling up production.
Another thing I see China doing, is pegging their currency to the dollar, now when the UK did this to the stronger German Deutsche Mark, the UK under Lord Norman Lamont crashed out of the Exchange Rate Mechanism and the UK saw interest rates hit 12/13% so I wonder if this chip shortage has more to it other than investment reasons only?
The lockdowns happened and some lever got pulled. They dusted off the “We love you America, come back and buy a car someday” commercial’s, cancelled everything, and opened the bottle of Scotch.
Perhaps this is an opportunity for FOSS silicon? Everybody understands that currently FOSS tooling/PDK's/etc isn't up to the task of creating a cutting edge CPU on a leading edge fab process, which is the usual retort when somebody brings up FOSS silicon.
But commodifying a sub-$1 chip? Yeah, why not?
You can buy used equipment from a few nodes back for basically scrap value, but building that machine today would cost more than building it when it was new.
You'd need to bootstrap a whole "old nodes" industry that does nothing but build obsolete machines to build obsolete chips once all the second/third/forth hand old node machines are gone.
In short: we've climbed the ladder and set it on fire behind us.
The risk really is entirely that we single source parts.
A few years back I designed something using only generic multi sourced parts and it was large and inferior. So there’s trade offs everywhere.
The point here would not be to replace traditional manufacturing but to augment it by acting as a backstop for total supply failure.
So traditionally, the chip industry worked such that that the high-end products moved first to a new process node, and less demanding equipment was fabbed on equipment a couple of generations old before that equipment was thrown on the scrap heap. Rinse and repeat, everybody was moving towards smaller and smaller nodes, just that some could trail the leaders by a few years.
But it seems these days the world is different, in that the cost per transistor has actually been going up for the leading edge processes, and power consumption has remained roughly flat. So for anything that doesn't need the latest gen process, there's really no upside to moving. Depending on who you ask, it seems the cheapest cost per transistor is somewhere in the 20-40nm ballpark. If so, it would imply there would be a long-time demand for these process nodes, and there is a need for an industry to supply tooling for these nodes, once the existing crop of second-hand equipment (left over when the leading edge users moved on to smaller nodes) is in need of replacement.
While correct, this is only true because of all the old infrastructure sitting out there that has been fully depreciated but still works fine. It used to be the latest and greatest thing and paid for itself back then. Nowadays it is just sitting on the asset sheet of a bunch of companies who are all going "if it still makes money, keep running it!".
The paradox of this all is what you identified: "if so, it would imply there would be a long-time demand for these process nodes". No one is going to invest in developing old nodes because the ROI isn't there. The demand is, but there probably isn't much cost differential between building a nearly leading edge fab and a 60nm fab.
If you could sell a turnkey 60nm fab right now that was delivered and setup quickly the demand would be insane. But no one can because fabs have never really been turnkey. When you factor in the cost to develop all of this, the capital you'd have tied up, etc. the ROI suddenly fails to compete with interest bearing bonds I'd guess.
Isn't this just because of overall chip shortage? We might see very different scenario if TSMC, Samsung and Intel overinvest now.
Also, you could say that cost per transistor increased for older processes as well...
>power consumption has remained roughly flat
Any source for this? Anything I've saw contradict that.
The new EUV steppers with soft x-ray wavelength can do in one shot what took 4 shots with 193nm ArF tools. And for each shot you have to apply, bake, develop inspect and strip the resist. This was supposed to help but the EUV steppers are so expensive it is kind of a wash price wise.
AFAIU, no. The basic problem, again AFAIU and I'm not an expert on this topic so take with a grain of salt, is that while the marginal cost of each transistor might still be less than at the previous generation, the R&D and capital costs have grown so immense that once you amortize those costs over the stuff produced for the lifetime of that process generation you end up with a higher per transistor cost compared to the previous generation.
> Any source for this? Anything I've saw contradict that.
Just search the web for articles about the breakdown of Dennard scaling, plenty of stuff to read on the topic. Even wikipedia mentions it https://en.wikipedia.org/wiki/Dennard_scaling
This is a really good description of what JIT does when no one considers economic reflexivity. The time duration for problems to surface is long enough that no one can "do the right thing" and last past next quarter's or next year's financial performance assessment (whether internally or by financial analysts). So everyone participates in a game theoretic sensible fashion, and gets burned either an SKU at a time as single source or limited source manufacturers burn out their legacy equipment, or all at once with black swans like this pandemic.
I'm certain no one who modeled JIT in business school research has worked out if it was more cash efficient to put up with and pay the piper when (not if) the disruptions happen, or to JIT only specific items that meet supply robustness criteria. The purchasing people mania for single sourcing plays into this dynamic. Supply chain management is enormously complex, but I fear they're still way behind the eight ball as they find out it gets fractally more complex than they had bargained for when they set out to cost rationalize it all.
Still not free though. I think people underestimate how niche all the skills are here.
I guess that's fine as long as we keep climbing, right?
For toilet paper, the cost of that disadvantage is having to wash your backside with the shower head and the benefit is the warm fuzzies of doing the right thing. For a business with no chips, it could be an existential threat. Warm fuzzies don't pay salaries.
The only way to avoid it is to have every company in the world to agree to not hoard, and harshly punish "defectors" somehow. Or have the supplier strictly ration supply, which doesn't work if the supply is coming from existing stock in the hands of distributors unless they also cooperate (or collude, depending on how you look at it).
None of that is remotely possible in the global industrial system. JIT is often blamed, but to avoid it, trillions of dollars worth of material would need to be hoarded constantly, which is its own problem: storage costs money, logistics and space, and its a substantial risk if you end up not needing it or the goods expire or become obsolete on a shelf and overproduction is a waste of resources. Going bust because you hoarded 5 million components that you never actually needed and couldn't offload is still going bust.
I prefer the bidet, and seeing normal reams of toliet paper cost now $30 for the "mega" value pack, I'm amused at the production of something, the trucking of something, the retailing of something that is ultimately discarded . The whole thinking and execution of it is backwards.
---
It is remotely possible, the governments can set immediate import/export restrictions and also guide manufacturing as they did during Covid for incubators and such. While it won't be effective, it can either help the stock/supply rebound, or throw into more fear...
At least in America, my Grandmother's generation (great depression) grew up with outhouses. When homes were renovated to add an indoor bathroom, they were done on the cheap.
Heck pre-pandemic the idea of a bidet was crazy, I had to work hard to find one that was reliable and at a good price. Now they are super available and advertised all over the place, but when I went to get one installed 10 years ago it was not easy just finding a reliable, cold water, model! Warm water models were obscenely priced in the US.
It was likely a chicken/egg thing. Consumers didn't know about them, so no one made them for the US market, which meant the few that did exist cost an arm and a leg, so no one bothered to buy them so no one bothered to mass produce them.
On top of that, when I inquired about getting an outlet installed behind my toilet for an electric bidet, my plumber and electrician thought I was crazy!
Same question in 2021, "sure we do that all the time".
Pretty sure mine didn't make them much money since they didn't include a pair of stainless steel mounting brackets, and they FedExed them to me. From Korea. In a box 100 times the necessary volume.
Bidets have gone mainstream in the US!
If the semiconductor shortage continues to worsen, I think we'll start to see calls for this type of intervention. One step that could be taken immediately is to prohibit sales from the distribution channel to overseas brokers, which are currently resulting in stupidity like $800 FPGAs that used to cost $20. But it's also true that this sort of market manipulation has downsides of its own.
I honestly don't know what the right thing to do here is. But I do know the industry had best unfuck itself, and that right soon, or this situation is going to have major consequences in everyday life.
is going to ? Already has.
Used car prices start climbing Jul '20.
That's got a huge ripple effect across the market, as best as I can tell, because it pushes people to buy older/lower cost cars across the board. Either that or they continue driving their old/potentially unsafe car even longer.
Basically even during the worst of the panic it only ran out for the duration it takes for a truck to be driven to the manufacturers warehouse, load it, drive to the shop and unload it.
With semiconductors we are actually running out of stock/capacity at the very beginning of the chain instead of last mile delivery.
I got all my electronic parts on time for my private development and first small manufacturing run. It is indeed no brainer.
This doesn't mean they should go out and build their own gas peaker plant.
We live in a society, and no man is an island. You can't control everything you depend on. Sure, you can control some things you depend on, but that's a long, and expensive rabbit hole. Hoarding twenty years of canned food and seventy thousand rounds of ammuniton is all well and good, unless doing so puts you in a situation where you can't pay rent.
That is because it's a utility that source power from several producers. The risk of no power what so ever for a prolonged time is not something to even be considered. Should that happen, society as we know it is probably at risk and my job is no longer important at all. Oh, and someone actually do plan for intermittent power delivery failure including having two different outside sources for power, a battery backup and diesel generators, data centers come to mind.
And if they intend to feed the generator in perpetuity, I'll happily shift the goalposts to some other critical supplier not providing their services.
Not going to ram 5V through a 7nm finFET.
Further, a lot of the designs for older processes have been iteratively improved (yield, performance, etc), but the older models (simulations) for them are insufficient to design new chips without explicit reuse or iteration. They are effectively stuck on that node at that foundry, unless they want to incur significant (opportunity) costs.
Everybody wanted small all-in-one devices last year, and nobody wanted big office printers. Companies like HP, Epson, etc won’t invest too much capital in production because that demand will implode later. Those companies probably sold 3-5 years of demand in a year for some SKUs.
It similar to the issue in the ports where everyone acts I’m their perceived best interest, which often screws over everyone.
My guess is that we’ll see a price collapse in the next 12-18 months for some commodities as the hoarding orders start getting fulfilled and companies are stuck with warehouses and balance sheets full of inventory.
I'd pay double right now for GPUs directly from the source, not from some sketchy third party.
Right now our EV/auto, space and even AR/XR industries as well as gaming and everything that requires chips, we are at the mercy of an external market that has a slant against the West. It will take some years to get out but we'll never not expect that in the future again. If costs go up costs go up, but availability should never be allowed to be used as leverage again, that is too risky and too costly long term.
Availability that is reliable is always more important than efficiency or cost, because right now lack of availability is costing lots of extra time that has the potential to lose entire industries, that is not acceptable in any way.
Very little margin and too much optimization/efficiency is bad for resilience. Couple that with private equity backed near entire market leverage monopolies/duopolies/oligopolies that control necessary supply and you have trouble.
HBS is even realizing too much optimization/efficiency is a bad thing. The slack/margin is squeezed out and with that, an ability to change vectors quickly. It is the large company/startup agility difference with the added weight of physical/expensive manufacturing.
The High Price of Efficiency, Our Obsession with Efficiency Is Destroying Our Resilience [1]
> Superefficient businesses create the potential for social disorder.
> A superefficient dominant model elevates the risk of catastrophic failure.
> If a system is highly efficient, odds are that efficient players will game it.
Hopefully that same mistake is not made in the future. It will take time to build up diversification of market leverage in terms of chips for availability. Hopefully we have learned our lesson about too much concentration, with that comes leverage and sometimes a "gaming" of the market.
This chip shortage, and all the supply chain problems during the pandemic as well, will hopefully introduce more wisdom and knowledge into business institutions that just because things are ok while being overly super efficient, that is almost a bigger risk than higher prices/costs. Competition is a leverage reducer. Margin is a softer ride even if the profit margins aren't as big.
This is why you see so many older and/or young-but-learned people telling you to read history - and yet, it's really hard to listen to them until you start getting personal, emotion-laden experiences that convince you to do so (such as this pandemic).
In a phone or laptop, obviously it would take too much physical space to have such components be changeable, but for vehicles and machinery couldn’t you regulate that common boards/chips/memory be easily accessible and interchangeable to a standard? (Kind of like how you have a fuse panel for your car or a fuse box for your house).
I get that this wouldn’t solve this issue, but it would certainly help by making parts that are susceptible to supply shocks be easily recoverable and reusable.
The problem is that the parts themselves aren't standardized. So often a very careful selection process followed by a requalification is required.
Why is that?
Also, you could use the design files from before any compensation you apply (to compensate for lithography errors), and use those files to derive a new compensation.
That is - timing closure is hard, and it's a "global" problem - if your design fails timing closure, you might have to re-do the entire layout and/or design itself.
However, if you change the gate capacitances, then I would expect that you'd just have to make a lot of small changes in a lot of different (but unrelated) places - which seems like, while tedious, would be immensely parallelizable with the number of engineers, and not risk any complete re-layouts like meeting timing closure would. So, a "redesign" more than a redesign.
What am I missing here?
- These legacy chips fabbed at legacy nodes are made with legacy equipment. In many cases, that equipment is a) fully depreciated and b) no longer available. Yes, most chips can be made with new equipment, but the profits with a new tool set are not going to justify the expense. You wouldn't buy a new $8 million CVD tool to make a 180 nm microcontroller that might sell for $2.
- Many chips are fabbed at 1 place only. The chip company owns the IP and has a defacto monopoly on the features of that chip, or even class of chips. If that fab is shut down, no more are being produced and customers have to buy up what brokers can scrounge. The more warehouses get picked out, the higher the prices go. In one extreme example, a chip made at Infineon's Kulim fab in Malaysia, that sold for $0.47, was purchased for $100 @ by (ironically) a chip equipment company.
- Hoarding has made the issue far worse. Companies are instructing buyers to order more than 1 year ahead, and seeing lead times exceed 1 year as well. The more these lead times extend, the farther into the future buyers extend their horizon and lead times just keep rolling.
- It may actually be the case that organic demand growth is not the main problem. More likely, it was the successful (and therefore vulnerable) just-in-time supply chain that collapsed as soon as COVID interrupted the choreographed flow of material.