Nvidia announces its most powerful AI chip as it seeks to become a platform co.

Re:Nvidia announces its most powerful AI chip as it seeks to... by azeirah on Monday March 18, 2024 @05:23PM (#39750208)

From how I understood it, it means they optimised the entire stack from CUDA to the networking interconnects specifically for data centers, meaning you get 30x more inference per dollar for a datacenter. This is probably not fluff, but it's only relevant for a very very specific use-case, ie enterprises with the money to buy a stack to serve thousands of users with LLMs.

It doesn't matter for anyone who's not microsoft, aws or openai or similar.

[ Parent | on HN ]

Re:Nvidia announces its most powerful AI chip as it seeks to... by misterdabb on Monday March 18, 2024 @07:39PM (#39751383)

It's a weird graph... It's specifically tokens per GPU but the x-axis is "interactivity per second", so the y-axis is including Blackwell being twice the size and also the increase from fp8 -> fp4, note it will needs to be counted multiple time as half as much data is needed to be going through the networks as well.

[ Parent | on HN ]

Re:Nvidia announces its most powerful AI chip as it seeks to... by acchow on Monday March 18, 2024 @06:10PM (#39750672)

They showed 30x was for FP4. Who is using FP4 in practice?

[ Parent | on HN ]

Re:Nvidia announces its most powerful AI chip as it seeks to... by KaoruAoiShiho on Monday March 18, 2024 @06:21PM (#39750768)

But maybe you should. Once the software stack is ready for it there'll be more people since the performance gains are so massive.

[ Parent | on HN ]

Re:Nvidia announces its most powerful AI chip as it seeks to... by dagmx on Monday March 18, 2024 @10:57PM (#39752509)

It would depend highly on the model though. Some stuff will generalize better to FP4 than others.

[ Parent | on HN ]