Release of Fugaku-LLM – a large language model trained on supercomputer Fugaku

Re:Release of Fugaku-LLM – a large language model trained on... by apsec112 on Monday May 13, 2024 @06:11PM (#40349123)

This honestly feels kind of silly. Back-of-the-envelope, this training run (13B model on 380B tokens) could be done in two months on a single 8x H100 node using off-the-shelf software, at a cost of around $35K from a cloud compute provider. They don't seem to list training time, but this cluster appears to use ~3.5 MW of power, so it's going to burn something like $500/hr just in electricity costs.

[ Parent | on HN ]

Re:Release of Fugaku-LLM – a large language model trained on... by LeoPanthera on Monday May 13, 2024 @06:47PM (#40349468)

But the whole point of this was to show that it could be done on CPUs during the worldwide shortage of GPUs.

Doing it in the CPU was the whole point.

[ Parent | on HN ]

Re:Release of Fugaku-LLM – a large language model trained on... by fnordpiglet on Monday May 13, 2024 @06:51PM (#40349501)

It could be done using abacuses too. A global shortage of GPUs is probably better addressed by producing more GPUs and making techniques more efficient.

[ Parent | on HN ]

Re:Release of Fugaku-LLM – a large language model trained on... by slt2021 on Monday May 13, 2024 @08:04PM (#40350064)

I would love to see human abacus calculating gradients like in 3 body problem https://i.imgur.com/pdz7xGh.jpeg

[ Parent | on HN ]

Re:Release of Fugaku-LLM – a large language model trained on... by fnordpiglet on Monday May 13, 2024 @11:58PM (#40351493)

Yes best scene on tv ever.

[ Parent | on HN ]

Re:Release of Fugaku-LLM – a large language model trained on... by Eisenstein on Monday May 13, 2024 @08:03PM (#40350055)

You are right. I assume that you have a fab ready to go and are licensed to produce CUDA capable GPUs?

[ Parent | on HN ]

Re:Release of Fugaku-LLM – a large language model trained on... by fnordpiglet on Monday May 13, 2024 @11:58PM (#40351494)

I’m certain they’re being built.

[ Parent | on HN ]

Re:Release of Fugaku-LLM – a large language model trained on... by rad_gruchalski on Tuesday May 14, 2024 @05:04AM (#40353091)

so it's being addressed and in the meantime someone has some fun

[ Parent | on HN ]

Re:Release of Fugaku-LLM – a large language model trained on... by fnordpiglet on Tuesday May 14, 2024 @08:24PM (#40361684)

Yeah man I’m down with using super computer clusters for anything. That’s just awesome. But the results seem more like marketing literature for GPU manufacturers to help rationalize more fab spend.

[ Parent | on HN ]

Re:Release of Fugaku-LLM – a large language model trained on... by mat_epice on Tuesday May 14, 2024 @01:46AM (#40352004)

The system uses 30MW, but this job used a portion of it that would consume 2.6MW.

There isn't really a figure for how much compute time it takes to train this thing, but 8x H100s have 32PF of AI compute among them. This job had 2,100 (half precision[1]) PF-in-fugaku / 158,956 nodes-in-fugaku * 13,824 nodes-in-job = 182 PF-in-job, implying it can get the job done 5.6x faster, or a little over ten days at the most optimistic.

Electricity costs for these nodes for ten days looks fairly similar to the rental costs of 8xH100s for 60 days according to my research. Lambda labs seems to have very cheap instances for 8xH100, but AWS and its ilk are much higher. However, the comparison is a little weird, as Fugaku is also a few years old now, and the contemporary GPU at the time of its release was the A100 (1/13th of an H100). The next Fujitsu chip may well narrow the power/performance gap between itself and (say) Blackwell or whatever is current at the time.

[1] https://www.fujitsu.com/global/about/innovation/fugaku/speci...

[ Parent | on HN ]

Re:Release of Fugaku-LLM – a large language model trained on... by numpad0 on Tuesday May 14, 2024 @09:56AM (#40355278)

I'm sure it's used for all sorts of industrial purposes, but Fugaku/K seemed a bit of solution in search of problem since NEC backed off(~2009). They were supposed to supply vector processing cluster interconnected with Fujitsu supplied scalar cluster that became K, but cancelled it stating vector processing is an outdated concept not worth pursuing during cash crunch the company was going through.

I can't seem to find a lot of technical documents regarding NEC SX processors online, but I could find outrageously priced PCIe accelerator cards(released 2021, $18k, 300W, 4.91TF FP32, 48GB VRAM) that are supposed to be its descendants, while Fugaku sits there GPU-less, maybe such is the life.

[ Parent | on HN ]

Re:Release of Fugaku-LLM – a large language model trained on... by Trapais on Monday May 20, 2024 @07:44AM (#40414424)

For comparison, here's 8B [Nemotron](https://huggingface.co/nvidia):

> 1,024 A100s were used for 19 days to train the model.

> NVIDIA models are trained on a diverse set of public and proprietary datasets. This model was trained on a dataset containing 3.8 Trillion tokens of text.

[ Parent | on HN ]