> Intel is well behind AMD and Nvidia in terms of GPU processing power and it might take them a while to catch up.

Have you read this article?

https://www.databricks.com/blog/llm-training-and-inference-i...

There I read that Intel Gaudi is behind all NVIDIA products in all metrics except performance per dollar.

Since one can only access these in the Intel cloud, where Intel has essentially made the few of these that exist freely available, isn’t perf per dollar kind of a joke metric? I mean, if they are free to use, they have infinite perf per dollar. But if Intel were to sell them, you were to buy them, and then you were to pay for their power, then wouldn’t perf per dollar look completely different ?

The article seems to try really hard to avoid saying how much perf per watt these have and how expensive they are. Reads a bit like a marketing piece as part of some Intel / data bricks collab (all companies and vendors do these).

> wouldn’t perf per dollar look completely different

Isn't the cost of actually running the card dwarfed by the cost of the card itself? e.g. even if we look at an old card like the A100 at 100% 24/7 that's about 3500 kWh per year which is what $300-$500 depending on the location?

if we Intel could come out with a card that could do anything the A100 does at half the price but 4x power usage it could still be somewhat competitive.

That’d depend on how many of these cards you need to get the same perf as an A100. Most of the results being shown are batch 1 (use the cards to serve 1 user). But in practice you’d use a single A100 to serve thousands of users concurrently (you can’t do that with the Gaudi 2 though).

The article measures that Gaudi 2 is competitive at “latency for batch size 1”, but that isn’t really a metric anyone cares about.

So Intel would need to sell these for much less than half, and then perf per watt would need to be much better (the article measures that it is currently worse than A100).

Comparing these things is hard, “if Intel were to sell these” is already speculation, since they aren’t on sale.

The article is right that perf per dollar is better, but that’s only because Intel is not making money out of these. As a user that’s a red flag, because if that continues these will be discontinued, and then any investment I do right now in supporting these in my SW stack goes to waste.

The 3060/4060 16GB is Nvidia's answer to "I'm a hobbyist/tiny business and I need an okay AI rig at a low price." If Intel could jam 32GB onto a card with comparable performance, and still compete on price, they might have a winner. They'd have my money, that's for sure.

I want to run Stable Diffusion at stupid resolutions, and I want it cheap.

Doesn't stable diffusion just start tiling things when you use large resolution sizes?

Not exactly tiled mind you, just "clearly repeating the same thing, with some variation".

Asking because when I was playing with it about some months ago, that's what it did. People explained (at the time) that it was because stable diffusion was trained on um... (from rough memory) 512x512 images.

It does do that but you can generate an image at the normal resolution and then scale it up to run more high-frequency sampling steps to add detail without losing the composition.

Ahhh cool. Sounds like I should muck around with it some more at some point then. :)

Not sure what you mean. I can tell you it pitches a fit if I tell it to go above ~1800x1800 on my 8GB card. I think automatic1111 has a flag to let it page stuff out to main RAM, but then everything is hideously slow.

k, sounds like things might have improved since I was last mucking around with that stuff (months ago). :)

I'm curious to know what prevents Intel putting 64GB or 128GB on a GPU - is there a technical limitation that makes it impossible to go to 128GB?

There would certainly be an awkward silence in response from Nvidia.

Given that Intel needs some angle to get ahead in the GPU wars I would have thought adding RAM would be an easy way to do that - especially given Nvidia and AMD seem to dish out RAM like its the most precious substance in the universe.

>especially given Nvidia and AMD seem to dish out RAM like its the most precious substance in the universe.

Well, it's only really nVidia. And the reason it's only nVidia doing this is because they don't want to self-cannibalize; if the 4080 had 24GB literally no prosumer or small organization wanting to play with the tech would even be considering the 4090 (and to a limited extent, the same is true of the 3080 vs. the 3090, but everyone was too busy mining Ethereum to notice that).

As far as the other manufacturers are concerned, there's no objective need for anything over 16GB on midrange cards (2070/3060/4060 equivalents) because that's the kind of GPU in the consoles and nobody releases exclusively for PC any more; you'll have to upgrade your card to have parity when the next generation comes out anyway so why provide more/drive costs up? It's not like AMD or Intel GPUs are in any way able to use that RAM for anything else, and why offer it now and kneecap themselves should they somehow get into a position where they actually can offer a good AI suite and want to sell 24GB cards for 2000+ dollars too?

And nVidia has taken full advantage of that fact, and that's why the 4080 only has 16GB even though the 7900XTX is performance-competitive with it and has the expected memory compliment. The 4080, including the Super, is a sub-par product because it kind of has to be.

The fact that Samsung and Hynix make it, maybe?

> especially given Nvidia and AMD seem to dribble out RAM like its the most precious substance in the universe.

Great quote, and apt description of this mildly infuriating trend.

Now I want to see a hardware mod that doubles the amount of RAM in a 4080.

As another commented, it's an easy way to enforce segmentation. So, not a good reason, but a reason.

I'd also imagine that the hardware on the smaller/cheaper dies doesn't have an address bus wide enough for 128GB.

> won't the AI folks all be loving it?

No, because that's not the only part of the system that matters. This indeed would be useful and Nvidia sure is pushing for more memory (note that they are also continuing development on GPUDirect), but there are limitations. Strapping a SSD onto the side isn't really going to make things any better.

Nvidia's secret sauce is CUDA. AMD has been doing great, but they still got a long way to go here. They caught up on hardware, but that's maybe half the battle? If it weren't, you'd see ML people flocking to AMD for lower prices and simply due to availability.

It's not that these companies are stingy with RAM, it is that GPU RAM isn't the same RAM you throw into your motherboard. They do have differences.

If it were easy, trust me, Nvidia and AMD would both pile on RAM and charge out the wazoo for it. I mean... look at the price difference between an A100 40GB and A100 80GB. They're the same chipset and there's a 2x price difference. Though I should mention the 80GB version is faster memory too.

And CUDA is not only C++, rather a quite mature polyglot ecosystem, with graphical debugging and plenty of frameworks to chose from.

Also to note, most folks that are driving ISO C++ heterogenous compute story are actually NVidia employees.

> If it were easy, trust me, Nvidia and AMD would both pile on RAM and charge out the wazoo for it. I mean... look at the price difference between an A100 40GB and A100 80GB. They're the same chipset and there's a 2x price difference. Though I should mention the 80GB version is faster memory too

No one nowadays throws a price based on costs + margin for R&D and little extra stay afloat. Those times seem to end somewhere early in 2000s. I wouldn't be surprised at all if that extra 40GB of faster RAM is in 30% cost increase range.

Cost matters.

> look at the price difference between an A100 40GB and A100 80GB

Nvidia is effectively a monopoly. They can charge whatever they want regardless of cost (which is what they are doing according to their reported margins)

Having a ton of VRAM doesn't really have much value without the speed to use it. After a point your calculation becomes heavy enough that you can get by with smaller VRAM and just accessing system memory, paying the data transfer or streaming cost.

GDDR6 doesn't support huge capacities. Maybe Intel could create an A770 32GB though.

What do you mean by "doesn't support huge capacities"? You can always add more channels in parallel.

Yeah, if you tape out a new chip; but that's a pretty vacuous statement. The capacities of GDDR6 modules available do impose a meaningful constraint on the total RAM that any given GPU can be equipped with, and while Intel is still struggling to get any foothold in the GPU market at all, it would be a big risk for them to invest in bringing to market a chip with eg. 4x the memory controller to compute ratio typically used by GPUs just so they could claim to have the cheapest 128GB GPU.

At some point you run out of pins, and the routing gets harder and harder. GDDR6 has very hard constraints on signal path length/quality. It gets expensive very quickly.

You also need memory bandwidth, that's harder than just quantity.

> even with Intel's underpowered GPU, won't the AI folks all be loving it?

No, they need to support at least pascal architecture.

> BUT - is there anything preventing Intel simply adding a ton of RAM to their cards?

The market isn't demanding it.

For gaming customers, modern games don't need more VRAM, 2k-4k textures can fit into 12-16GB of VRAM and 8k textures mostly don't improve the look at a 4k monitor resolution, unless these textures are stretched over large objects. With upscaling technology, Nvidia customers aren't running at 4k native resolution, so 2k textures do fine for 1080p upscaled to 1440p or 1440p upscaled to 4k. Gaming customers want raster performance primarily, assuming VRAM is above 10-12GB.

We will see retail demand for more VRAM when monitor resolutions exceed 4k, and this won't happen on a 1-3 year timeframe.

For AI customers, there's already specialized cards like H100.

> For AI customers, there's already specialized cards like H100

I don't know anything about machine learning but I think that is the question. Isn't nVidia professional tier like the H100 rare and expensive? I remember someone here on HN started a sharing club where they said they'd pool resources together to buy these things and allow people to basically timeshare or rent them?

Is this like the situation where nVidia is basically writing your code for you by providing abstractions like it does with nVidia hair works and things like that on the gaming side? Would Intel have to basically develop its own answer to CUDA or could it gang up with AMD on this?

Yes H100s are very expensive (think ~$40k per unit), and very hard to obtain. For this reason they're basically all sent to cloud operators who can sign large bulk deals up front and then rent them out. At that point rubber really hits the road because it's where the hardware is basically auctioned off on an hour by hour basis, so you can easily pay $40k per month for access to the cards because there's so much money flooding into AI right now.

This is why OpenAI seem to have burned through 10s of billions of dollars of Azure compute credits in, like, a few years.

Yes CUDA is basically a giant developer platform made by NVIDIA for their cards. It's libraries, tools, languages, tutorials, and other ecosystem stuff like the fact that lots of people use it. Intel and AMD might or might not need their own answer to it. They need at least parts of an answer but CUDA is a lot more than AI, so if that's all you care about, you can skip parts of it.

Now AMD has been trying to develop a CUDA competitor for a long time but it's clear either their heart isn't in it, or they just really struggle to attract skilled dev plaform people. ROCm seems to suck so hard people would rather pay NVIDIA's monopoly pricing than deal with it, which is impressively bad.

Of course the reason NVIDIA is in this position is that they supported all kinds of R&D with their CUDA effort, and it turned out to be AI that hit big. If you just implements the bits of CUDA that seem most useful you'll always be behind.

The H100 is orders of magnitude too expensive for 99% of people who wants to do AI inference.

> For AI customers, there's already specialized cards like H100.

Which is extremely expensive because Nvidia can charger whatever they want because they effectively are a monopoly. Since power or per card performance doesn't really matter that much yet Intel could in theory compete only on price as long as they manage to cobble something at least semi-decent together.

Intel has been behind discrete GPUs for decades, even when they thought Larrabee would be a better alternative.

While they might score on Linux land by open sourcing their drivers, long time ago their OpenGL drivers weren't that great either.

> Intel has been behind discrete GPUs for decades

They only entered the discrete GPU market a couple of years ago to be fair.

Fortunely I am old enough to remember what a couple of years ago means, like 1998.

https://en.wikipedia.org/wiki/List_of_Intel_graphics_process...

So the last time they released a non integrated GPU prior to 2022 was back in 1998? That seems hardly relevant..

That's Apple Silicon territory.

It's always amusing how all these LLM or diffusion models work on some beefy Nvidia GPU's or Macbook Air M1's with 16GB of RAM because all that RAM can be utilised by the GPU or the Neural Engine.