Jensen revealed later that the LLM inference is 30x due to architectural improvements, it's massive. I don't know if it's latency or just 2-3x performance boost with 30x more customers served in the same chip. Either way, 30x is massive.

He always does that. They stack up a bunch of special case features like sparsity that most people don't use in practice to get these unrealistic numbers. It'll be faster, certainly, but 30x will only be achievable in very special cases I'm sure.

Isn't sparsity almost always a win at this point? Making everything fully connected is a major waste.

The kind of sparsity that the hardware supports is not fully general. I'm not aware of any large models trained using it. Maybe they are all leaving 2x perf on the table for no reason, but maybe not. I don't think sparsity is really proven to be "almost always a win" for training.

To train well with it I think you still need to store all the optimizer state (derivatives and momentum or whatever) if not all the weights (for RigL), so maybe not nearly as much memory bandwidth advantage as you get in inference?

The other big announcement here is NIM - Nvidia Inference Microservice.

It's basically TensorRT-LLM + Triton Inference Server + pre-build of models to TensorRT-LLM engines + packaging + what appears to be an OpenAI compatible API router in front of all of it + other "enterprise" management and deployment tools.

This software stack is extremely performant and very flexible, I've noted here before it's what many large-scale hosted inference providers are already using (Amazon, Cloudflare, Mistral, etc).

From the article:

'Nvidia will work with AI companies like Microsoft or Hugging Face to ensure their AI models are tuned to run on all compatible Nvidia chips. Then, using a NIM, developers can efficiently run the model on their own servers or cloud-based Nvidia servers without a lengthy configuration process.

“In my code, where I was calling into OpenAI, I will replace one line of code to point it to this NIM that I got from Nvidia instead,” Das said.'

The dead giveaway is "I changed one line of code in my OpenAI code" which means "I pointed the OpenAI API base URL to an OpenAI compatible API proxy that likely interfaces with Triton on the backend via its gRPC protocol".

I have a lot of experience with TensorRT-LLM + Triton and have been working on a highly performant rust-based open source project for the OpenAI compatible API and routing portion[0].

On this hardware (FP4) with this software package 30x compared to other solutions (who knows what - base transformers?) on Hopper seems possible. TensorRT-LLM and Triton can already do FP8 on Hopper and as noted the performance is impressive.

[0] - https://github.com/toverainc/ai-router

30x is the type of number that when you see it in a generational improvement, you should ignore it as marketing fluff.

From how I understood it, it means they optimised the entire stack from CUDA to the networking interconnects specifically for data centers, meaning you get 30x more inference per dollar for a datacenter. This is probably not fluff, but it's only relevant for a very very specific use-case, ie enterprises with the money to buy a stack to serve thousands of users with LLMs.

It doesn't matter for anyone who's not microsoft, aws or openai or similar.

It's a weird graph... It's specifically tokens per GPU but the x-axis is "interactivity per second", so the y-axis is including Blackwell being twice the size and also the increase from fp8 -> fp4, note it will needs to be counted multiple time as half as much data is needed to be going through the networks as well.

They showed 30x was for FP4. Who is using FP4 in practice?

But maybe you should. Once the software stack is ready for it there'll be more people since the performance gains are so massive.

It would depend highly on the model though. Some stuff will generalize better to FP4 than others.

Yeah and the 30x is largely due to the increase in factors like packaging and throughput. It's not indicative of general purpose performance which is what I was talking about.

Again, I do think the throughput and energy efficiency gains are impressive, but the raw performance gain is lower than I'd have expected for the massive leap in node size etc

The 30x number is for a really narrow scenario tbh. Running a GPT 1.8T parameters (w/ MOE) on one GB200

'narrow scenario,' perhaps, but one that also happens to closely match rumors for GPT4's size

But Blackwell in the graph is FP4 whereas Hopper is FP8.

This is also the only place Nvidia are getting competitive pressure - from the likes of Groq (and likely but less published from Cerebras) with higher inferance T/s and concurrency utilization/batching [1] so if this proves to be the true then the case for big chip systems (on todays specs) will be harder.

[1]https://twitter.com/swyx/status/1760065636410274162?t=rpbcr8...

How is 2.5x disappointing in one generation?

Did you skip the sentence immediately after that one?

It’s two fused chips. So 1.25x per chip. 25% uplift. Not 2.5x uplift. The 2.5x is for the whole package.

Is that how it works? Why don't we just put many chips in one computer?

the massive blackwell SoC he showed is two Blackwell dies with an interconnect. It’s very similar to what Apple does with their Ultra series.

Then the B200 package is 2 of these plus a CPU. So a total of 4 GPU dies in each unit.

> Then the B200 package is 2 of these plus a CPU.

That's GB200.

> two fused chips

Jensen's comment about being first was such a dig to Emerald Rapids.

Is it the first? The Apple Ultra series chips are two Max’s fused with an interconnect. In which case it’s both CPU and GPU.

I believe this is just the first for a GPU only product.

Compare to the 10x that was Hopper uplift.

Because it involved scaling in chip area needed for FP8. AI community realized that FP8 training is possible few years back so the transistors given for FP8 was scaled. Overall I think transistors grow just by ~50% per generation so most of the gains comes from removing FP32/FP64 share which were dominant 10 years back, but there is only some point it could go to.