News for hackers. Stuff that matters.
Re:Microsoft introduces its own chips for AI, with eye on cost by gary_0 on Wednesday November 15, 2023 @12:51PM (#38279847)
> Microsoft said it does not plan to sell the chips

Add it to the list of things you can't buy at any price, and can only rent. That list is getting pretty long, especially if you count "any electronic device you can't fully control or modify".

Re:Microsoft introduces its own chips for AI, with eye on cost by ronsor on Wednesday November 15, 2023 @01:09PM (#38280093)
Google does the same thing with their TPUs. The masses will be left with the NVidia monopoly, while large companies will be able to free themselves from that.
Re:Microsoft introduces its own chips for AI, with eye on cost by aleph_minus_one on Wednesday November 15, 2023 @01:40PM (#38280556)
> The masses will be left with the NVidia monopoly, while large companies will be able to free themselves from that.

My bet: if it really becomes clear what capabilities an AI accelerator chip needs and lots of people want to run AIs on their own computers, AI accelerators will appear at the market. This is how capitalism typically works.

My further bet: these AI accelerators will initially come from China.

Just look at the history of Bitcoin: initially the blocks were mined on CPUs, but then the miners switched to GPUs and "everybody" was complaining about increasing GPU prices because of all those Bitcoin mining. At some moment Bitcoin mining ASICs came from China and after those spread, GPUs were not attractive anymore for Bitcoin mining (of course the cryptocurrency fans who bought the GPUs for mining attempted to use their investment for mining other cryptocurrencies).

Re:Microsoft introduces its own chips for AI, with eye on cost by brucethemoose2 on Wednesday November 15, 2023 @03:31PM (#38282017)
The capital costs are enormous, not even counting the CUDA moat. It takes years to start producing a big AI processor.

Yet many startups and existing designers anticipated this demand correctly, years in advance, and they are all still kinda struggling. Nvidia is massively supply constrained. AI customers would be buying up MI250s, CS-2s, IPUs, Tenstorrent accelerators, Gaudi 2s and so on en masse if they wanted to... But they are not, and its not going to get any easier once the supply catches up.

Unless there's a big one in stealth mode, I think we are stuck with the hardware companies we have.

Re:Microsoft introduces its own chips for AI, with eye on cost by solardev on Wednesday November 15, 2023 @04:00PM (#38282416)
Is there not a distributed computing potential here like there was for crypto mining? Some sort of seti@home/boinc like setup where home users can donate or sell compute time?
Re:Microsoft introduces its own chips for AI, with eye on cost by fragmede on Wednesday November 15, 2023 @04:08PM (#38282506)
you can setup a computer and sell time on it on a couple of saas platforms, but only for inference. for training, the slowness of the interconnect between nodes become a bottleneck.
Re:Microsoft introduces its own chips for AI, with eye on cost by solardev on Wednesday November 15, 2023 @04:09PM (#38282531)
I see, thanks for the explanation!
Re:Microsoft introduces its own chips for AI, with eye on cost by latchkey on Wednesday November 15, 2023 @04:08PM (#38282504)
The capex/opex is different for ML/AI than it was for crypto mining. Totally different hardware profiles.
Re:Microsoft introduces its own chips for AI, with eye on cost by brucethemoose2 on Wednesday November 15, 2023 @04:43PM (#38282975)
Yes, see projects like the AI Horde and Petals. I highly recommend the Horde in particular.

Theres also some kind of actual AI crypto project that I wouldn't touch with a 10 foot pole.

But ultimately, even if true distribution like Petals figures out the inefficiency (and thats hard), it had the same issue as non Nvidia hardware: its not turnkey.

Re:Microsoft introduces its own chips for AI, with eye on cost by aleph_minus_one on Wednesday November 15, 2023 @04:16PM (#38282620)
> Yet many startups and existing designers anticipated this demand correctly, years in advance, and they are all still kinda struggling.

As I already hinted in my post: I see a huge problem in the fact that in my opinion it still is not completely clear to this day which capabilities an AI accelerator really needs - too much is in my opinion still in a state of flux.

Re:Microsoft introduces its own chips for AI, with eye on cost by brucethemoose2 on Wednesday November 15, 2023 @04:47PM (#38283020)
The answer is kinda "whatever Nvidia implements." Research papers literally build around their hardware capabilities.

A good example of this is Intel canceling, and AMD sidelining, their unified memory CPU/GPU chips for AI. They are super useful!.. In theory. But actually, they totally useless because no one is programming frameworks with unified memory SoCs in mind, as Nvidia does not make something like that.

Re:Microsoft introduces its own chips for AI, with eye on cost by jasonwatkinspdx on Thursday November 16, 2023 @01:57AM (#38286492)
There's also potential IP issues.
Re:Microsoft introduces its own chips for AI, with eye on cost by abstractcontrol on Thursday November 16, 2023 @05:05AM (#38287669)
> Yet many startups and existing designers anticipated this demand correctly, years in advance, and they are all still kinda struggling. Nvidia is massively supply constrained. AI customers would be buying up MI250s, CS-2s, IPUs, Tenstorrent accelerators, Gaudi 2s and so on en masse if they wanted to... But they are not, and its not going to get any easier once the supply catches up.

Can you order any of these devices online as a regular person? Anybody can order a $300 Nvidia GPU and program it. This is the reason why deep learning originated on the GPUs. Forget those other AI accelerators, even if you bought something like a consumer grade AMD GPU, you couldn't program it because it's restricted. The reason why Nvidia's competitors are struggling is because their hardware is either too expensive or hard to buy.

Re:Microsoft introduces its own chips for AI, with eye on cost by tomcam on Wednesday November 15, 2023 @04:02PM (#38282433)
> My bet: if it really becomes clear what capabilities an AI accelerator chip needs and lots of people want to run (or even train) AIs on their own computers, AI accelerators will appear at the market.

My bet: in 6 months jart will have models running on local or server, with support for all platforms and using only 88K of ram ;)

Re:Microsoft introduces its own chips for AI, with eye on cost by llm_nerd on Wednesday November 15, 2023 @01:32PM (#38280432)
nvidia is a $1.2 trillion dollar company (the 6th largest company by cap), and at this point AI is a huge component of that wealth. It has appreciated by 3.3x since just the beginning of this year.

If any of these companies truly made competitive silicon they absolutely would commercialize it.

I suspect they aren't as competitive as the press releases hold them to be, and this Microsoft entrant is likely to follow the same path. Like Google, Tesla, Amazon and others it seems mostly an initiative to negotiate discounts from nvidia.

It would be great if there were really competition. When Google was hyped about their Tensor chips they did have a period where they were looking to commercialize it, and there are some pretty crappy USB products they sell.

Re:Microsoft introduces its own chips for AI, with eye on cost by jsnell on Wednesday November 15, 2023 @02:24PM (#38281169)
They are commercializing the silicon, by selling access to it on their clouds.

Now, I know that what you actually mean is selling the chips themselves to third parties :) But it's not obvious that there's any point to it given their already existing model of commercializing the chips.

First, literally everyone is already supply-constrained due to limits on high end foundry capacity. Nvidia has a ton of capacity because they're one of TSMC's top two customers. The big tech companies will have much smaller allocations which are used up just supplying their own clouds. Even if the demand for buying these chips rather than renting were there, they just don't have the chips to sell without losing out on the customers who want to rent capacity.

Second, the chips by themselves are probably not all that useful. A lot of the benefit is coming from the silicon/system/software co-design. (E.g. the TPUv4 papers spent as much attention on the optical interconnect as the chips). Selling just chips or accelerator cards wouldn't do much good to any customers. Nor can they just trust that systems integrators could buy the cards and build good systems to house them in. They need to sell and support massive large scale custom systems to third parties. That's not a core competency for any of them, it'll take years to build up that org if you start now. And it means they need to ship the software to the customers, it can't continue being the secret sauce any more.

Nvidia on the other hand has been building up an ecosystem and organization for exactly this for the last decade.

Re:Microsoft introduces its own chips for AI, with eye on cost by scarface_74 on Wednesday November 15, 2023 @03:33PM (#38282038)
> Nvidia has a ton of capacity because they're one of TSMC's top two customers. The big tech companies will have much smaller allocations which are used up just supplying their own clouds.

And TSMCs top customer is not even playing in the cloud space.

Re:Microsoft introduces its own chips for AI, with eye on cost by wmf on Wednesday November 15, 2023 @01:17PM (#38280227)
MI300 is coming.
Re:Microsoft introduces its own chips for AI, with eye on cost by machinekob on Wednesday November 15, 2023 @01:54PM (#38280743)
This time AMD for sure will fight with NV (its only failed 20 times already copium)
Re:Microsoft introduces its own chips for AI, with eye on cost by wmf on Wednesday November 15, 2023 @02:05PM (#38280910)
On one hand this is a fair prediction but Triton exists now and it didn't exist last time.
Re:Microsoft introduces its own chips for AI, with eye on cost by pjmlp on Thursday November 16, 2023 @02:23AM (#38286636)
If the software still sucks with what CUDA can do, in graphics tooling, polyglot support and optmized libraries, across at very least Windows and GNU/Linux, than whatever the card is capable of doing isn't much relevant.
Re:Microsoft introduces its own chips for AI, with eye on cost by FuriouslyAdrift on Wednesday November 15, 2023 @01:57PM (#38280793)
Re:Microsoft introduces its own chips for AI, with eye on cost by layer8 on Wednesday November 15, 2023 @01:14PM (#38280180)
Sounds like an opportunity for hardware startups.
Re:Microsoft introduces its own chips for AI, with eye on cost by wmf on Wednesday November 15, 2023 @01:16PM (#38280222)
There are over a dozen AI hardware startups.
Re:Microsoft introduces its own chips for AI, with eye on cost by layer8 on Wednesday November 15, 2023 @01:42PM (#38280582)
Sounds like they are taking the opportunity.
Re:Microsoft introduces its own chips for AI, with eye on cost by Tempest1981 on Wednesday November 15, 2023 @01:55PM (#38280763)
Waiting to be assimilated by the big guys
Re:Microsoft introduces its own chips for AI, with eye on cost by scarface_74 on Wednesday November 15, 2023 @03:34PM (#38282059)
Hardware startups aren’t going to stand a chance because they have to fight for the scraps of capacity that is left over after Apple and NVidia and the cloud providers use what they can.
Re:Microsoft introduces its own chips for AI, with eye on cost by cjdrake on Wednesday November 15, 2023 @04:16PM (#38282608)
Re:Microsoft introduces its own chips for AI, with eye on cost by er4hn on Wednesday November 15, 2023 @05:16PM (#38283371)
This is a custom chip that they are making. I don't think that they should be required to sell it, but if others find it valuable you could expect to see hardware startups making their own RISC-V AI chips as well that you could buy.
Re:Microsoft introduces its own chips for AI, with eye on cost by pdq on Wednesday November 15, 2023 @06:26PM (#38283888)
It's the new HaaS business model: Hardware as a Service.
Re:Microsoft introduces its own chips for AI, with eye on cost by chongli on Wednesday November 15, 2023 @07:06PM (#38284195)
I’m curious to know what everyone thinks of this trend. Do you view it as a good thing, bringing efficiency and economy of scale, competition and so forth? Or do you consider it a bad thing, another salvo in the War on General Purpose Computing [1] so vividly described by Cory Doctorow?

I, personally, am interested in retrocomputing, amateur/hobbyist electronics, and hobbyist computing (including semiconductors [2]). While these techniquess and devices may be light years away from anything resembling a computer that can compete with SotA commercial offerings, they do offer the promise of “keeping the candle lit” as it were. I will note that if you follow Sam Zeloof’s chronicles, he progressed through the earliest phases of semiconductor development far faster than the industry did back when it was pioneering the technology. Of course, he had the benefits of knowing it was already possible and access to the written knowledge of the experts who went before him.

[1] https://boingboing.net/2012/01/10/lockdown.html

[2] https://www.youtube.com/watch?v=qCSIGejNT4M

Re:Microsoft introduces its own chips for AI, with eye on cost by userbinator on Wednesday November 15, 2023 @09:33PM (#38285279)
IMHO definitely a bad thing; and beyond the war on general-purpose computing, it's the "you will own nothing and be happy" war on ownership.
Re:Microsoft introduces its own chips for AI, with eye on cost by pjmlp on Thursday November 16, 2023 @02:24AM (#38286646)
Nah, it is still the same old business only rebranded for new folks, timesharing systems.