I would say software is very important. We already have tons of different models, standards, library, etc. I usually have smooth experience if I am using NVDA, but any other variation, I have to spend some times to get things started.

Supporting the common librarires that I use is very important for me to chose the cloud platform.

Considering that Jensen is on stage with Satya at the moment sharing the keynote of Microsoft Ignite, I suspect NVIDIA won't be going anywhere anytime soon.

A bit surprised to see Jensen's stage appearance since clearly Microsoft's success with its own AI chips means less business for Nvidia's chips.

Because different than the ARM chip also announced in the same Ignite event, Microsoft doesn't exactly "need" nor can fully utilize an AI chip. Google trains its foundational models (e.g. Gemini) on its own TPU hardware but Microsoft's is heavily reliant on OpenAI for its generative AI serving needs.

Unless Microsoft is planning to acquire OpenAI fully and switch over from Nvidia hardware...

> Unless Microsoft is planning to acquire OpenAI fully

They're going to play a modified version of the old Rareware trick.

It's also a pretty great game to buy up OpenAI equity, which ultimately gets spent on Microsoft compute. Two birds, one stone.

I don’t think the margins here are so considerable that we can assume revenue = profit

They can be losing money, but gaining in market share and moat.

Almost nobody in this game cares about profit right now.

...they're going to overbid on a studio that was actively falling apart, after being rebuffed from buying one of the biggest giants in the business[0], all as part of an ill-advised attempt to muscle into a game business they didn't understand?

[0] Microsoft tried to buy Nintendo very early on

Microsoft absolutely runs their own models on their own hardware, at scale, and they have done so for years just like every other hyperscaler -- Project Brainwave was first publicly talked about as far back as 2018. The generative LLM craze is a recent phenomenon in comparison. They are absolutely going to go all in on putting AI functionality in Bing, in Excel, in Windows, etc etc. To do that, you need hardware.

None of this is really strange. It also wasn't strange when Google announced H100 systems while also pushing TPUs they developed. Microsoft has Jensen on stage because customers of Microsoft Azure demand Nvidia products. Customers of Google Cloud demand Nvidia products. So, they provide them those products, because not providing them loses those customers. It's that simple. Everyone involved in these deals acknowledges this.

Awesome! Someone who knows about Brainwave!

Yeah! Did/do you work on it? The original publications were good timing; I was working as a consultant on an FPGA-based ML accelerator at the time the original stuff was talked about, and I really enjoyed reading everything I could about Brainwave! Really neat project from both a system design perspective (e.g. heterogeneous compiler) to the choice of using and interconnecting FPGAs and integrating the network/software/ML stack (IIRC, there was a good paper on the overlay network they used to make those custom functions available on the global network fabric.)

I'm guessing at this point the ASICs make a lot more economic sense, though. :)

Sorta! Tangentially on the data science side - Brainwave pre-dates me being at MS though. It's a really cool project and I'm glad someone else also thinks so :) I was enamored with FPGAs from undergrad, one of the professors I did research with was a consultant on the original Catapult project, I thought it was the coolest idea. It was pretty awesome to be able to come full circle a fews years later.

They’ve been doing custom accelerators for Bing for a few years. See: project catapult (2012). It’s all on FPGA, but similar.

In some ways, Microsoft was 10 years ahead, but they are terrible as an organization at proliferating research projects to production across multiple orgs.

nVidia still has the monopoly on training. Everyone else is just making chips for inference.

While NVIDIA provides the best absolute performance for training, Intel (i.e. Gaudi) already provides much better training performance per dollar.

The funny thing is that this fact has been shown inadvertently by NVIDIA:

https://www.servethehome.com/nvidia-shows-intel-gaudi2-is-4x...

That article shows that it takes about 50x as long to train gpt-3 with intel's offering vs Nvidia. At least in the current environment, if you are training llms I think almost no amount of cost savings can justify that.

That 50X is only if you can afford one thousand NVIDIA H100.

There cannot be more than a handful of companies in the entire world that could afford such a huge price (tens of millions of $).

In comparison with a still extremely expensive cluster of 64 NVIDIA H100, the difference in speed would reduce to only two to three times, and paying several times less for the entire training becomes very attractive.

You’ve got a really warped sense of scale if you think only a handful of companies couldn’t afford tens of millions of dollars.

That’s complete pocket change for any of the Fortune 500.

The problem is not having so much money available.

Such a big expense only makes sense for a company where spending that amount would bring hundreds of millions of $ of additional revenue.

I doubt that any of the companies that have already spent such amounts have recovered even a small part of their expenses. It is more likely that they bet on future revenues, but it remains to be seen who will succeed to achieve that.

Kinda, companies of that scale regularly spend more than that on (often) random R&D.

Sure if there is a plausible ROI, they’d have no issues dropping that much money (actually far more). Revenues for fortune 500’s are going to be in the 10’s of billions anyway, and it wouldn’t be hard to make an argument that random AI project could increase that by a couple percent or decrease costs a couple percent, which would more than provide that ROI.

Their biggest issue is usually having anyone in leadership that has a clue enough to even propose something plausible, let alone get a team together to give it a plausible go.

If they have that, Capital is not the issue.

huggingface supports multi-node training?? https://huggingface.co/docs/optimum/habana/usage_guides/mult...

am i missing something here? just like you'd want to scale an h100 cluster out beyond one box of 8, you'd do the same for gaudi2?

Yeah no one really cares about this for large models. Iteration speed matters, and no one is waiting 50 times as long to train a LLM.

Intel has always published the "training per dollar" because no one else competes.

Even for fine tuning you are almost always better off getting smaller GPU cloud instances.

Amazon announced Trainum last year and recently announced their partnership with Anthropic, where Claude will be trained on Trainium.

https://press.aboutamazon.com/2023/9/amazon-and-anthropic-an...

Shouldn't inference be 99.999% of the compute cost over time? Especially for MS. Look how many Copilots they are cramming into their products

It's probably 25-75 for inference vs training. I estimate 10K tokens/month * 100M users/month = 1T tokens of inference/month. Training is 2-3 times more expensive, the training set is 15T tokens, so about equivalent to 45T tokens of inference. But the lifetime of a model ~1 year, they spend 45T tokens to train and 12T tokens for inference. So the inference cost is about 21% of the total cost.

This estimate is based on a conservative 10K tokens/user/month. In reality it could be much higher if they send PDFs or other input documents. That might increase the ratio maybe to 50-50, but in any estimate it's not 100:1.

The Maia chip Microsoft announced it's also for training.

> Amazon announced theirs last year

Inferentia (inf1) was GA'ed in December 2019 so it's actually almost 4 years old now. The trainium (trn1) chips and the Inferentia 2 (inf2) refresh is indeed 1 year old though.

I was referring to trainium.

Yikes those names are horrible.

They'll sound better after we hear Microsoft's names.

Microsoft Azure Inference for Cloud Apps 365 Pro Live Series X

I think you mean

Microsoft Azure™ Inference for Cloud Apps© 365 Pro® Live Series X™®

... For Workgroups

64-bit for Extended 64-bit systems.

Viva

I'm still so bitter about this nonsense https://www.microsoft.com/en-us/security/business/identity-a...

> Azure Active Directory is now Microsoft Entra ID

ok, geez, thanks

I don’t really know what an active directory is, but I assume that the default type of directory is a passive one, in that it just holds files or subdirectory (it doesn’t act). An active directory sounds like a directory that is going to play tricks on me.

Entra ID sounds like a type of ID.

I’m not sure how something could legitimately have each of these names. I assume the functionality changed pretty dramatically over the lifespan of the product?

Active Directory is Microsoft's database targeted mostly for user login and associated data.

That’s kind of a nice take actually. For people like me who did Windows systems engineering at the beginning of the millennium, Active Directory is a very household name. But there’s not much actual meaning behind it. Everything Microsoft was active in that era - Active Directory, Active Desktop, ActiveX, Active Server Pages (lives on as ASP.NET) and probably some more I forgot.

I should admit that I was somewhat lying, I’ve encountered Active Directory before and know it is something vaguely identity/login related.

I was pretty confused the first time I encountered it, though. Could tell from context that it had something to do with accounts, but thought maybe it could be for syncing a user’s home directory or something!

It actually isn’t a terrible name, in isolation, since a directory (like, the non-digital version) was for keeping identities. But “directory” in tech has a pretty strong association with file systems.

The term "directory" was actually already quite common for these kinds of systems (directory services) when AD came about, which is probably why it was called Active Directory. The Active bit was some kind of "en vogue" term for Microsoft back then, as mentioned ;)

It technically builts upon (amongst other systems) LDAP, the Lightweight Directory Access Protocol and X.500

I guess you could think of it more like a telephone directory - which again is also where the file system metaphor has its roots I guess. So the two are not so different in the end.

I manage our small company’s Microsoft 365 / Azure tenant and only rarely need to use the portal.

A month or so ago my laptop was requiring a BitLocker recovery code before it would boot. I spent ages looking for Azure AD, before eventually discovering it was renamed to Entra. They should have had a transition period where it would have “ (formerly called Azure Active Directory)” on its name.

Why? Inferentia => inference, trainium => training. Given the usually naming of AWS product, having one where the name roughly matches what it does is pretty good?

TPU is pretty good but is associated with Google. MTIA is an acronym but still maps to what the chip does. ~~"Cobalt" is worse as it does not mean anything~~ . Cobalt is the CPU chip, MAIA is the accelerator so this matches Meta's naming.

> Why? Inferentia => inference, trainium => training.

Funny, that's precisely why I think the names are bad. It's like if Google had chosen "Search-ola" as their name. Way too on the nose and/or lazy. Having said that, I don't really care all that much and I imagine that may have been the spirit of those who chose the names.

heh, as someone who has to deal with this nonsense all day <https://aws.amazon.com/products/> I would for sure welcome some straightforward naming. $(echo "AWS Fargate" | sed s/Fargate/ServerlessContainerium/)

My previous role was a lot of AWS, and I became convinced that the value of an AWS cert was mostly learning how to map all of the product names to their actual functions.

Hahaha, it might not have worked but I would love to use a search engine called Search-ola

They're searchable and self-explanatory so they're not bad.

Graviton CPU is a year older.

If they “can’t make the chips fast enough” being TSMC’s second highest volume customer behind Apple and probably second in priority, what chance does Microsoft have getting enough of TSMCs capacity?

Microsoft does have the benefit of not having any customers other than themselves, so volumes are smaller.

They're more constrained by advanced packaging (CoWoS) capacity rather than the manufacturing of the silicon.

But from the pictures, the Maia 100 is also using CoWoS packaging. That will be necessary for any new HBM chip.

The latest Stratechery interview had Pat Gelsinger on it and he was talking about how Intel is happy to package up TSMC dies in their foundries. Relevant quote:

> So it really brings together many of the things that Intel is doing as an IDM, now bringing it together in a heterogeneous environment where we’re taking TSMC dies. We’re going to be using other foundries in the industry, we’re standardizing that with UCIe. So I really see ourself as the front end of this multi-chip chiplet world doing so in the Intel way, standardizing it for the industry’s participation with UCIE, and then just winning a better technology.

> what chance does Microsoft have getting enough of TSMCs capacity?

Looking at their balance sheet: plenty.

TSMC is all about pay to play. Apple is first in line because they’re willing to pay to be first in line. I have no doubt Microsoft can justify spending the money they save not buying nvidia into getting some priority access from TSMC.

Also keep in mind Apple is now on 3nm so there’s likely spare 5nm.

In exchange for paying to build out TSMC's 3mm capacity, Apple gets exclusive rights to use it for the next N years. Who knows what other exclusivity deals or other shenanigans are in place that would prevent Microsoft from acquiring capacity?

My guess is that Microsoft booked capacity years ago. Either by strategic decision, with the belief they could yield the capacity to someone else, or through their other hardware divisions (see: Xbox).

It’s very likely they knew something like this was coming, as they’ve been doing FPGAs for more than a decade now.

It blows my mind that not only are Apple, Nvidia, and all other enormous "chip makers" fabless... but there's ONE single company actually making the majority of all their chips.

Why aren't any other companies entering this space? TSMC's growth and profits are immense, it's not like the market couldn't bear more competitors.

Also, the current situation is geopolitical insanity. It feels like China and the U.S. are on a path to war in the next decade or so. China is itching to retake Taiwan... and if you thought the U.S. fought a lot of wars over oil, there's NO WAY we wouldn't go to war to prevent all the world's most important semiconductors falling exclusively under PROC control. It's the 21st century, we would have no choice. I know that TSMC is diversifying and building fabs in the U.S. and Germany, and hopefully that will reduce the risk of war. But it's just nuts that this is even a risk. How does one company control so much of the global market?

TSMC is solving some of the hardest physics, electrical engineering, mechanical engineering, and other kinds of engineering I don't even know about. It's unlikely anyone catches up to them in terms of lithography ability for a long, long time given how well they've executed 45nm and lower nodes.

More like ASML is solving the physics problems and TSMC is the one with the most capital and experience to run their EUV machines. It’s not that other competitors don’t exist, it’s more like they’ve given up or can’t keep up. Global foundations I believe isn’t making the investment past 16nm. Samsungs latest node doesn’t perform as good as TSMC and intel is being intel.

> Global foundations I believe isn’t making the investment past 16nm.

GlobalFoundries' Fab 1 can do 1 nm, and its Fab 8 is capable of doing 14 nm:

> https://en.wikipedia.org/w/index.php?title=GlobalFoundries&o...

1 nm is very impressive.

I meant 12 nm. Sorry for the typo. :-(

No. I don’t know where this oft-repeated semiconductor meme came from, but it’s a really poor take.

ASML are solving a small subset of physics problems: how to project extremely small features. TSMC are solving many more physics problems: how to structure layers of doped silicon into transistors, how to structure those transistors into logic gates, and those logic gates into functional blocks. This is why silicon process is not just a matter of capital investment, and why nobody is going to show up overnight with 10 billion dollars and change things. It’s not that TSMC are the only ones with a big bag of cash to give ASML.

Yeah I love how people thinks it's easy to mass produce designs that take hundreds of individual steps and weeks for a single wafer to go through the process. 50+ Layers, aligned at the (sub?) nanometer level. Trying to tweak the process so that your electrons stop quantum tunneling across layers in all directions.

A fab is far, far, more than ASML's machines. A chip can take around 3 months going through various steps repeatedly.

ASML's machines are very impressive, but they're just one piece of that puzzle. And ASML themselves rely on collaborators like Zeiss.

This idea that it's all just capital and others are reluctant to invest otherwise they could duplicate TSMC simply false.

TSMC makes a lot of money from fabricating chips on old technology using fully depreciated assets. No new competitor is going to spend billions today to fab chips on older technology.

It's insanely hard.

TSMC is so dominant because they have very, very, deep institutional knowledge. Even if you have billions to invest it's not easy to create that, and definitely impossible to do it short term.

There's only 3 companies left chasing smaller nodes. Everyone else has given up and are focusing on revenue from high volume parts in older processes.

The only three are Samsung, Intel, and TSMC. Idk why going on with Intel but Samsung should BSs doing better but that institutional knowledge that TSMC has is crazy. They all technically invest in asml that makes the machines. If I had to redo my education I’d get into electrical engineering just for this.

> I’d get into electrical engineering just for this

I get the impression it's mostly physics guys.

Everything is physics when you get down to the fundamentals.

So "Chemistry, the central science" did not convince you?

> Everything is physics when you get down to the fundamentals.

Relevant xkcd: https://xkcd.com/435/

TLDR: Math is not. :-)

Nah, neither physics nor electrical engineering. It's all chemistry at TSMC.

Eh, the process engineers I've talked to don't have a great life, at least here in the US. It seems inevitable you end up perpetually on call for some machine, and I don't mean some type of machine, I mean unit #23 over in fab #4 and then that's your baby for years. The pay is considerable worse than software too.

I almost went to get a CE degree because I find it all so fascinating but ultimately I'm glad I didn't.

If you want the best people to work in an area you need to pay them. I'm surprised given the possible returns these aren't seven figure jobs.

the pay is worse than software because you're competing with Asian wages in a manufacturing context, where the margins are slim.

the serious system architects, designers, and fab managers are doing just fine, software or not. the dudes in Phoenix who are getting pushed through 2-year degrees to help run a clean room are probably making 65k.

If enough funding magically appeared, GlobalFoundries could be a contender, right?

> If I had to redo my education I’d get into electrical engineering just for this.

This is the crazy thing. Such fundamental important technology but salaries generally suck. Much more money in cat memes and influencer channels.

Yeah the salaries for EEs are really depressing in most of the industry. It's really hard work. Most EEs end up leaving for software or research (I did!).

I took EECS, did a masters in analog design and I've only worked professionally making shitty SaaS websites. Making way more money with way less stress while still feeling accomplished that I learned how to do the complex stuff I wanted to know how. Very hard to make nearly as much otherwise.

I went to school for CS while one of my best friends went for EE. It definitely seemed like the far more complex discipline.

In fact for a small project in an elective he programmed a computer vision system that was far ahead of anything me or my peers had made at the time in our CS program. He had pretty much zero programming experience.

I must admit he's a particularly smart person but it was crazy that I went on to make more money than him. His work is way more complicated than mine as far as I can tell.

There is one more player joining that small club, Huawei

I was under the impression they had their chips manufactured by SMIC.

Exactly, the fourth player quickly catching up to the others is SMIC.

If there's one company that has the capability to become a Samsung/Intel (IDM) on steroids, it is Huawei. They have the money,talent, determination and state backing.

Presumably only for their own and Chinese brands however as I can't see anyone letting them do fabication for major players outside of China. They've also got a huge hurdle given they cant buy the standard equipment every chip fab needs owing to it all being produced by a single company.

Why doesn't anyone enter? It takes billions and billions of dollars, tons of knowledge and if you get it wrong...even Intel screwed it up, and they are one of the only companies would could even try to do it.

Right now Intel is hamstrung because there is no separation between the fab and design sides. Nobody is going to trust Intel to fab their latest and greatest while the two sides are joined at the hip.

The sooner Intel spins out the fab side into its own entity the better positioned it will be to pick up the business that TSMC doesn't have capacity for.

I've seen plenty of Amazon's competitors running on AWS, plenty of Google's competitors using gmail, and suchlike.

People will overlook a lot, if the product is compelling enough and the sales guys can keep a straight face while promising not to peek.

But the whole problem at Intel is that they're not at the level of TSMC, they're years and generations behind. Splitting them up won't just magically make the fab business be better, and without the profit margins of the sale of processors it's unlikely they'll have the resources to invest to get them to be good. Case in point: AMD are doing very well after spinning off their fabs, but their fabs are not doing very well (relatively of course - they're just stuck to where they split and haven't had meaningful advances; but they're still a profitable business because those older chips are needed too).

> It takes billions and billions of dollars

Apple and Microsoft are trillion dollar companies ( both nearing $3 trillion market caps ). They can afford it. Heck they have the balance sheet to acquire both tsmc and samsung if either were up for sale.

I thought Apple was moving towards being a full-stack company. Where all the hardware and software is developed in-house. If anyone has the resources, it is apple.

> I thought Apple was moving towards being a full-stack company. Where all the hardware and software is developed in-house. If anyone has the resources, it is apple.

Having a modern chip fab binds an insane amount of capital, and makes you much more vulnerable in case of market turbulences. This is exactly a reason why there exist people who like to claim that Intel should spin off their fabs.

Depends how those market turbulences occur, for example a Chinese invasion of Taiwan will be a disaster for Apple.

It would be value destructive for Apple to buy TSMC and not service any other customers

Building and operating a fab is an extremely difficult and expensive task that will take at least 5-10 years to bear fruit. That makes it a high-risk venture which is hard to sell to the board and investors.

More importantly, the new chip production lines are dependent on hyper-specialized EUV lithography systems that only company is able to manufacture (ASML). ASML has their own limits to production too.

Enter? Companies are trying to keep up.

Just few years ago was four companies doing the bleeding edge: Intel, GlobalFoundries, Samsung and TSMC. Intel was the best. First Global Foundries could not keep up. Then Intel fumbled ant TSMC became the best. Samsung is the only one keeping up with TSMC but they come slightly behind. Intel tries to catch up the two. No companies have "exited" they just can't compete.

The bleeding edge semiconductor node design is like doing Apollo moon program every 4-5 years. Only one can be the best.

>Why aren't any other companies entering this space?

There are? INTC and Samsung?

>falling exclusively under PROC control

It wouldnt fall under PROC control.

It would be destroyed.

Plus I don’t even think PRoC would know how to operate the EUV machines.

The USA would certainly bomb it if they had to

TSMC fabs have bombs build in to destroy everything if PROC invades I've read.

That's a bit hyperbolic don't you think? Sure, they're the only game in town when it comes to the leading edge nodes (eg. 3nm), but there's plenty of options at 14 nm and above. That corresponds to AMD zen (released in 2017) and Intel skylake (released in 2015). They're significantly worse than the latest and greatest CPUs today, but they're still in use today. I'd hardly call that as "going back to the stone age".

You read bullshit.

I also have a bridge I can sell you. And trust me: you've never seen one like this, at rates like I'm offering! Buy now!

And, If china bombs ASML we'll be back to stone age.

How would China do that exactly? None of their bombers have sufficient range.

They have intercontinental ballistic missiles. If needed, they can be retrofitted with non-nuclear payload.

China launching any kind of missile at ASML is WW3

The Chinese aren't stupid enough to do that. Any launch of ICBMs at a NATO country would be interpreted as a possible nuclear attack and draw an immediate US counterforce strike.

No, you'll just have to learn how to write software again.

>it's not like the market couldn't bear more competitors.

The market could only bear if they are willing to pay a lot more in the name of diversification. Otherwise most would simply have trouble keeping up.

We really need some basic FAQ on HN for hardware technology related topic.

It’s not for a lack of trying on Samsung’s and intel’s part that tsmc is dominant ;)

It's worse. It's one company in a 'contested' territory. So much so that US military is rumored to have considered the 'samson option' of bombing the TSMC facility were the PLA to occupy the island.

Taiwan (unlike Ukraine) is far too critical to the world economy.

By internal and external claims, Intel is on track to be competitive with TSMC by 2025. And, Nvidia and other will use the capacity as soon it is available.

> retake Taiwan

Take Taiwan. You can’t “retake” something you never had.

And TSMC uses the ASML fabrication machines made by… one company. Basically two places are responsible for all of those chips. Intel really needs to get back in the game. Hopefully that’s still on the table.

Even if Intel could spin up a bunch of fabs competitive with tsmc, they'd still be competing for asml's capacity. We just can't massively scale up leading edge fab capacity.

> Why aren't any other companies entering this space?

Because it would take over a decade to build a comparable chip fab and thats assuming you had unlimited cash to throw at it.

You'd be spending billions with zero revenue for a decade and no guarantees it'll be anything as close to as good as TSMC's setup. Try finding someone willing to invest in that and it becomes clear why theres not more chip fabs out there.

> It feels like China and the U.S. are on a path to war in the next decade or so

IMO, MAD (mutually assured destruction - if a nuclear power start a war with another nuclear power, both will probably end up destroyed by nukes) makes the prospects of that highly unlikely.

> It feels like China and the U.S. are on a path to war in the next decade or so

they are literally having discussions in San Fran right now to reduce the likelihood of that happening. And it sounds like China is quite open to discussion, and the US as well.

Well. China is promising things and they'll ignore it as usual until they are sure to have the upperhand.

They are building navy fleet like no other.

Still harassing neighbors ( eg. Philippines).

Nothing changed. They just want a bigger advantage.

They buy capacity that is little behind and optimized differently. Apple and Nvidia buy the bleeding edge and develop it with TSCM.

Azure Maia is TSMC N5, I think.

Nvidia H200 and H100 are TSMC 4N.

Intel Foundry apparently has some room.

> Amazon announced theirs last year

Amazon has had Graviton chips (after they acquired Annapurna Labs) since 2018. Or do you mean specifically AI-oriented chips?