News for hackers. Stuff that matters.
Re:Maintaining large-scale AI capacity at Meta by chaos_emergent on Sunday June 16, 2024 @09:44PM (#40701608)
When training a large model you're doing a single forward pass + backprop over multiple Infiniband-connected nodes for a single model instance, so if one node goes down it takes a logical unit of nodes down with it. For reference, GPT-4 was rumored to be around 1.7T, and doing some back-of-the-hand math[1], that's like 500-700 H100 GPUs per model instance, which means you need a multiple of that for any training parallelism whatsoever.

[1] back-of-the-hand-math: 1.7T * 4 bytes = 6.8 TB; 3-4x that for activation + gradients = 27.2 TB; 27.2TB / (80GB / H100) = 349 H100s; 1.5-2x conservative multiplier accounting for not fully using node resources + memory overhead in the machine = ~500-700 H100s.

truly insane numbers.

Re:Maintaining large-scale AI capacity at Meta by furyofantares on Sunday June 16, 2024 @10:12PM (#40701747)
That trillion+ parameter count is the sum of each of the "experts", right?
Re:Maintaining large-scale AI capacity at Meta by oersted on Monday June 17, 2024 @03:23AM (#40703059)
The ever-circulating rumour is 1.7T - 1.8T for the whole thing. But it is not very substantiated, mostly started by SemiAnalysis and geohot based on rather loose speculation (such as API latency and price), and not much solid evidence to confirm it after that.

And of course, it must have changed substantially with GPT-4-Turbo and GPT-4o. It would make sense if the cost reduction was larger than the price reduction, they probably have a higher profit margin now, and the price reduction has been very significant since GPT-4 release.