News for hackers. Stuff that matters.
Re:ChatGPT Hardware a Look at 8x Nvidia A100 Powering the Tool by Const-me on Tuesday February 14, 2023 @10:41AM (#34790893)
A100 has 1555 GB/sec of memory bandwidth. 50% more than Radeon 7900 XTX, but the AMD GPU costs 10x less.

nVidia’s top management did amazing job with their strategy. They have spent decades, and millions of dollars, developing and promoting CUDA. Now this motivates people to spend that amount of money on nVidia’s hardware.

I don’t believe the trend continues for ML inference, at least not for long. Unlike traditional HPC stuff like FEM or fluid dynamics, the GPU code of ML inference is not that complicated. The complexity is limited to the data being processed, these ML models.

It’s not terribly hard to port the inference to other hardware architecture or software stack, reusing the models. I think it’s only a matter of time until AI companies like OpenAI will start porting their code from CUDA to VulkanCompute or some other vendor-agnostic stack, to avoid buying or renting such $100k computers. Don’t get me wrong, these computers are awesome, just very expensive.

Re:ChatGPT Hardware a Look at 8x Nvidia A100 Powering the Tool by mambru on Tuesday February 14, 2023 @11:33AM (#34791807)
You should probably compare the H100:

https://www.techpowerup.com/gpu-specs/h100-sxm5-96-gb.c3974

With the MI300, which has 128GB and more compute power (at least on paper):

https://www.techpowerup.com/gpu-specs/radeon-instinct-mi300....

A100 can be compared with MI250, perhaps. It also has 128GB.

Re:ChatGPT Hardware a Look at 8x Nvidia A100 Powering the Tool by Patrick-STH on Tuesday February 14, 2023 @12:14PM (#34792526)
The MI300 will compete more with Grace Hopper than with the H100 SXM5. We will be covering and showing both later this year.
Re:ChatGPT Hardware a Look at 8x Nvidia A100 Powering the Tool by metadat on Tuesday February 14, 2023 @04:38PM (#34796448)
What is the pricing like for the MI300's?
Re:ChatGPT Hardware a Look at 8x Nvidia A100 Powering the Tool by BizarroLand on Tuesday February 14, 2023 @05:48PM (#34797349)
I would guess $12k-$16k, but they haven't released the price yet.
Re:ChatGPT Hardware a Look at 8x Nvidia A100 Powering the Tool by mhio on Tuesday February 14, 2023 @05:16PM (#34796971)
An Azure 8x MI250X OAM enclosure was out and about at the SC22 conference late last year (second half of the linked article).

https://www.servethehome.com/microsoft-azure-at-sc22-and-the...

Re:ChatGPT Hardware a Look at 8x Nvidia A100 Powering the Tool by ericpauley on Tuesday February 14, 2023 @10:46AM (#34790994)
There's also a problem of dataset size. the 7900XTX maxes out at 24GB when a lot of new models require the higher-end 80GB A100 variants. The A100 is also a server-grade card, so even spec-for-spec it's not a fair comparison.
Re:ChatGPT Hardware a Look at 8x Nvidia A100 Powering the Tool by Const-me on Tuesday February 14, 2023 @11:05AM (#34791339)
> a lot of new models require the higher-end 80GB A100 variants

For training, yeah. But at least in some cases, inference can be optimized to use less memory.

For their large Whisper model, OpenAI says they need 10GB of VRAM: https://github.com/openai/whisper#available-models-and-langu...

My DirectCompute port of that uses slightly over 4GB of VRAM, see the last column in that table: https://github.com/Const-me/Whisper/blob/master/SampleClips/... I haven’t actually optimized memory usage I only did the DirectCompute port, the memory savings were achieved by Georgi Gerganov in whisper.cpp project, which I ported to Windows.

Re:ChatGPT Hardware a Look at 8x Nvidia A100 Powering the Tool by anentropic on Tuesday February 14, 2023 @11:10AM (#34791425)
apparently GPT-J needs ~25GB RAM min (for inference only), i.e. a 32GB card ...and GPT-3 is an order of magnitude bigger again

https://nlpcloud.com/gpt-3-open-source-alternatives-gpt-j-gp...

though TBH just making cards with more RAM to fit these kind of models seems easier than being at absolute cutting edge of performance

I also wonder how this metric translates to something like Apple Silicon 'unified memory'... latest MBP have 32GB RAM, I wonder if anyone managed to run GPT-J there yet?

Re:ChatGPT Hardware a Look at 8x Nvidia A100 Powering the Tool by anentropic on Tuesday February 14, 2023 @11:17AM (#34791550)
There's a float16 version of GPT-J rewritten in C++ here that apparently fits into 16GB...

https://github.com/ggerganov/ggml/tree/master/examples/gpt-j

it's CPU-only and runs inference at "about ~6 words per second" on an M1 MBP

Re:ChatGPT Hardware a Look at 8x Nvidia A100 Powering the Tool by smoldesu on Tuesday February 14, 2023 @12:02PM (#34792322)
That's for the max-size model (6B parameters). GPT-Neo has pruned models 125m and 1.3b that fit into 1.5gb and 6gb of memory respectively.

These models are obviously nerfed, but you can run them on budget ARM hardware and get legible, paragraph-length responses in less than 10 seconds.

Re:ChatGPT Hardware a Look at 8x Nvidia A100 Powering the Tool by flangola7 on Tuesday February 14, 2023 @05:15PM (#34796955)
There's a 20B model now
Re:ChatGPT Hardware a Look at 8x Nvidia A100 Powering the Tool by smoldesu on Tuesday February 14, 2023 @05:39PM (#34797253)
The 20B model was the original I believe, and the other models were all pruned from it.
Re:ChatGPT Hardware a Look at 8x Nvidia A100 Powering the Tool by atylerrice on Tuesday February 14, 2023 @10:01PM (#34799505)
This is incorrect. Thought would be interesting. Pruning usually removes some parameters from the model (setting certain ones to 0 for instance). The smaller models all have less layers and therefore weren’t pruned but were smaller to begin with.
Re:ChatGPT Hardware a Look at 8x Nvidia A100 Powering the Tool by mark_l_watson on Tuesday February 14, 2023 @02:41PM (#34794813)
They have a link to the bottom for something similar, but with GPT-2. The Hugging Face docs have hints for using FP16, and avoiding double memory requirements for model and checkpoint data.
Re:ChatGPT Hardware a Look at 8x Nvidia A100 Powering the Tool by atylerrice on Tuesday February 14, 2023 @10:03PM (#34799522)
You can also use int8 quantization which comes for free with huggingface and isn’t a huge cost to accuracy. Which cuts the memory usage almost in half again. (almost in half because some layers might stay 32 bit if they make a tiny part of the model)
Re:ChatGPT Hardware a Look at 8x Nvidia A100 Powering the Tool by trifurcate on Tuesday February 14, 2023 @01:04PM (#34793308)
I am running GPT-J-6B inference on 13GB of VRAM. FP16 is a must at this scale. I've also loaded GPT-NeoX-20B using int8 quantization onto a single RTX 3090 before (24GB VRAM) however inference latency was quite high.
Re:ChatGPT Hardware a Look at 8x Nvidia A100 Powering the Tool by andruby on Wednesday February 15, 2023 @04:48AM (#34801810)
Actually, the MacBook Pro with M2 Max CPU can be configured with 96GB of ram. The Mac Studio with M1 Ultra can have up to 128GB of ram.
Re:ChatGPT Hardware a Look at 8x Nvidia A100 Powering the Tool by anentropic on Wednesday February 15, 2023 @05:17AM (#34801974)
Ah great, I hadn't seen that... seems like that should be plenty for some of these models!
Re:ChatGPT Hardware a Look at 8x Nvidia A100 Powering the Tool by TOMDM on Tuesday February 14, 2023 @01:14PM (#34793480)
I've been running the stock implementation that OpenAI put out, I didn't know people had made so many improvements!

Thank you for your work.

Re:ChatGPT Hardware a Look at 8x Nvidia A100 Powering the Tool by godelski on Tuesday February 14, 2023 @01:22PM (#34793629)
As someone who trains large models, this is pretty much the only number I care about. Batching saves more time, and gets better results, than clock speed.

For the A100, A6000s, and H100s you are paying for the memory and the upcharge for being a server card. AMD isn't really competing in this space, but they are moving in that direction if you look at the exascale machines[0,1]

[0] https://www.top500.org/system/180047/

[1] https://www.amd.com/en/products/server-accelerators/instinct...

Re:ChatGPT Hardware a Look at 8x Nvidia A100 Powering the Tool by erichocean on Tuesday February 14, 2023 @07:09PM (#34798153)
> Batching saves more time, and gets better results, than clock speed.

How much time is saved?

If you've got a 12GB card at 5x speed vs. a 96GB card at 1x speed (so 5x slower), is the former still competitive? How much faster does the 12GB card need to be to get the same quality results with less memory (and smaller batches)?

What if the faster card has 24GB of memory?

Re:ChatGPT Hardware a Look at 8x Nvidia A100 Powering the Tool by atylerrice on Tuesday February 14, 2023 @09:59PM (#34799479)
Well the batch size also helps with making training more stable. So a larger batch size computes the gradient across more examples so it ends up being more stable than only using a batch size of day one. A batch size of one is only fixing the error on one example rather than many.

a larger Batch size can help keep the gpu fed but usually that’s not a problem with these larger models.

Re:ChatGPT Hardware a Look at 8x Nvidia A100 Powering the Tool by godelski on Wednesday February 15, 2023 @05:30AM (#34802064)
Hard to say tbh. Could be infinite, could be zero.

In classification batch is going to help a ton (including stability) and you may not even get top results without a high batch, so infinite in that case. More realistically, consider that batch also helps with GPU I/O. I/O slows things down quite a lot, so if you're moving stuff in and out of the accelerator a lot then you can easily be I/O bound.

For generation the conversation gets more complicated. For GANs the smaller card could do better if you can fit a proper batch size there (probably not tbh but often closer to 16 or 24 so that may be better here). Here distributed tends to help more. For VAEs, Diffusion, Autoregressive, etc are more stable and can also hugely benefit from large batches.

So the truth is an unsatisfying "it depends." I know that may not be the answer you're looking for, but it is the real one. It takes a lot of work and tricks to scale (as anyone working in non-ML HPC will tell you very similar stories about scaling). But also note that the clock speeds aren't as big of a gap as you note though the memory gaps are.

Re:ChatGPT Hardware a Look at 8x Nvidia A100 Powering the Tool by TallGuyShort on Tuesday February 14, 2023 @02:25PM (#34794610)
Could you educate me on what's really different about a card that is "server-grade" vs. one that is not?
Re:ChatGPT Hardware a Look at 8x Nvidia A100 Powering the Tool by jmalicki on Tuesday February 14, 2023 @02:40PM (#34794795)
Two of the biggest differences are ECC memory and virtualization support.
Re:ChatGPT Hardware a Look at 8x Nvidia A100 Powering the Tool by crabbone on Tuesday February 14, 2023 @04:26PM (#34796274)
Yeah, also while ECC is mostly transparent to the software layer, virtualization support is in itself a huge software layer that comes with this kind of equipment. MIG ( https://docs.nvidia.com/datacenter/tesla/mig-user-guide/ ), there's also software made to expose GPU inside containers.

This software is mostly supplied by vendor, and it mostly cares about vendor's tech, and will usually require special features exposed by the hardware, so, won't work with consumer-grade GPUs.

Re:ChatGPT Hardware a Look at 8x Nvidia A100 Powering the Tool by m463 on Tuesday February 14, 2023 @04:50PM (#34796633)
well, and it's a specialized die. A100 ~ 2x size of 3090
Re:ChatGPT Hardware a Look at 8x Nvidia A100 Powering the Tool by ericpauley on Tuesday February 14, 2023 @03:39PM (#34795621)
For these GPUs, it partially comes down to form factor (being able to drop 8 in a chassis with unified cooling). I’d say that’s the primary difference here. This applies maybe a little more noticeably to the rest of the computer around the a100s, with ECC ram, hot swappable components, and redundant power supplies. The 8x a100 rigs are absolute beasts, difficult to move even by two people without removing some components. There is also price discrimination here where manufacturers aim for higher margins on server components (as nvidia clearly does).

Between the two mentioned cards, though, it really does come down to performance. The cards are simply not interchangeable for large scale model training.

Re:ChatGPT Hardware a Look at 8x Nvidia A100 Powering the Tool by synergy20 on Tuesday February 14, 2023 @10:52AM (#34791095)
indeed the CUDA software is the moat for Nvidia, though its hardware is still great.
Re:ChatGPT Hardware a Look at 8x Nvidia A100 Powering the Tool by fancyfredbot on Tuesday February 14, 2023 @02:42PM (#34794817)
At the scale of investment OpenAI are making the software shouldn't be the moat anymore - you can hire a lot of very good software engineers with ten percent of that hardware budget.
Re:ChatGPT Hardware a Look at 8x Nvidia A100 Powering the Tool by synergy20 on Tuesday February 14, 2023 @04:14PM (#34796092)
it's way more than that, building an ecosystem takes time, like a decade or more, lots of tooling, libraries, optimizations etc have to be done, and they have to be done in a way users think it's either similar to CUDA, or is simpler, the moat is very wide and deep as far as I can tell.
Re:ChatGPT Hardware a Look at 8x Nvidia A100 Powering the Tool by fancyfredbot on Tuesday February 14, 2023 @04:26PM (#34796271)
I feel like very few ML researchers actually use CUDA. You'd likely get a long way with an optimised backend for XLA to cover tf/jax, probably something similar for pytorch. That's a lot of work but not sure it's a whole decade.
Re:ChatGPT Hardware a Look at 8x Nvidia A100 Powering the Tool by synergy20 on Tuesday February 14, 2023 @05:09PM (#34796879)
Not sure if that's entirely tree. CUDA has 85% or more of the market share last time I checked, intel's OneAPP and AMD's ROMA? are so behind that made their Machine learning hardware totally uninteresting for ML related purchases, sadly.
Re:ChatGPT Hardware a Look at 8x Nvidia A100 Powering the Tool by Const-me on Tuesday February 14, 2023 @05:15PM (#34796965)
I don’t think OpenAI needs a new ecosystem. I think they just need a cost-efficient way to run inference of their ML models, for large userbase.

That smaller task is way more manageable. It shouldn’t take a decade, maybe a month or two. Vulkan ain’t some new tech, the API arrived years ago and is widely used by game developers.

Thanks to Valve work on DXVK 2.0, there’s even an implementation of MS DirectCompute GPU API over Vulkan on Linux. I’m not saying doing that is necessarily the best approach for the job, just one of the many options.

Re:ChatGPT Hardware a Look at 8x Nvidia A100 Powering the Tool by cma on Tuesday February 14, 2023 @11:12AM (#34791467)
With nvidia's sparsity support memory bandwidth can be effectively doubled for inference, if you can sparsify.
Re:ChatGPT Hardware a Look at 8x Nvidia A100 Powering the Tool by 2OEH8eoCRo0 on Tuesday February 14, 2023 @02:03PM (#34794282)
> It’s not terribly hard to port the inference to other hardware architecture or software stack

Sounds like a job for AI. Universal translator and all that.

Re:ChatGPT Hardware a Look at 8x Nvidia A100 Powering the Tool by 6gvONxR4sf7o on Tuesday February 14, 2023 @02:49PM (#34794928)
> 50% more than Radeon 7900 XTX, but the AMD GPU costs 10x less

You could say that about some of nvidia's other GPUs too. A better comparison would be AMD's server offerings, which are priced comparably.

> Unlike traditional HPC stuff like FEM or fluid dynamics, the GPU code of ML inference is not that complicated.

It's complicated enough relative to the problem ML people are trying to solve. That's why in ML, all these tools have emerged specifically to let practitioners not care about the GPU code. People writing pytorch code don't want to think about CUDA vs vulkan at all. It's more of a question for the people writing the foundational libraries like pytorch.

I sincerely hope that tools like openai's triton make it easier to support a variety of GPUs.

Re:ChatGPT Hardware a Look at 8x Nvidia A100 Powering the Tool by Const-me on Tuesday February 14, 2023 @03:25PM (#34795446)
> You could say that about some of nvidia's other GPUs too.

Technically, yes. Practically, nVidia forbids to deploy their other GPUs in data centers, in the EULA of their drivers: https://www.datacenterdynamics.com/en/news/nvidia-updates-ge...

May or may not be enforceable, probably depends on jurisdiction, but I’m pretty sure that EULA gonna scare some people.

> People writing pytorch code don't want to think about CUDA vs vulkan at all.

Well, there’re also other people. OpenAI’s CEO doesn’t necessarily write pytorch code, but he described the compute costs of ChatGPT as “eye-watering”.

> I sincerely hope that tools like openai's triton make it easier to support a variety of GPUs.

I’ll believe it when I see it. Performance portability across GPU architectures ain’t trivial. For instance, in my Whisper implementation, the most expensive compute shader is the one which computes product of 2 matrices.

nVidia version: https://github.com/Const-me/Whisper/blob/master/ComputeShade...

AMD version: https://github.com/Const-me/Whisper/blob/master/ComputeShade...

These two compute shaders have little in common, they even use different memory layout for the arguments, despite both compute shaders are doing the same math.

Re:ChatGPT Hardware a Look at 8x Nvidia A100 Powering the Tool by GistNoesis on Tuesday February 14, 2023 @03:05PM (#34795174)
This TB/sec of memory bandwidth is only when the data is on the GPU. The real problem is that GPU memory is expensive and models are getting bigger.

What we would like to do is what has usually been done with the cache hierarchy where you bring the data as needed to the GPU. As this would allow to only need a lot less memory, as we could stream parameters from RAM or disk as needed.

But there is a bottleneck at the pcie interface ( 32GB/s for pcie 4.0x16 ), and you can't feed the GPU fast enough. Which means you can't leverage cheap memory directly.

Re:ChatGPT Hardware a Look at 8x Nvidia A100 Powering the Tool by irusensei on Wednesday February 15, 2023 @02:58AM (#34801217)
What are the chances someone pulls a Toy Story and Nvidia gets SGI’d by clusters of consumer grade graphic cards? Knowing greedy nvidia they’d probably try to nerf their gaming cards for that task but its not like they are a monopoly.
Re:ChatGPT Hardware a Look at 8x Nvidia A100 Powering the Tool by Const-me on Wednesday February 15, 2023 @06:05AM (#34802313)
I don’t know what the chances are, but over time it might happen. Economy of scale should not be underestimated.

30 years ago, there was a consensus x86 architecture was only good for small computers, while servers need faster specially-designed processors like IBM Power or Dec Alpha. Then AMD64 was introduced, after a while they took server market too.

15 years ago, Intel was making specialized HPC co-processors called Xeon Phi. The chips were actually good, exceeded 1 TFlop of FP64, and weren’t terribly hard to program. I think it were consumer-grade graphic cards which made it obsolete. Despite much harder to program, the raw performance numbers were better, and cost efficiency in flops/$ was an order of magnitude better.