How Meta trains large language models at scale

Re:How Meta trains large language models at scale by jeffbee on Wednesday June 12, 2024 @10:12PM (#40665267)

I don't see how you can evaluate better and worse for training without doing so on cost basis. If it costs less and eventually finishes then it's better.

[ Parent | on HN ]

Re:How Meta trains large language models at scale by tmostak on Wednesday June 12, 2024 @11:18PM (#40665594)

This assumes that you can linearly scale up the number of TPUs to get equal performance to Nvidia cards for less cost. Like most things distributed, this is unlikely to be the case.

[ Parent | on HN ]

Re:How Meta trains large language models at scale by logicchains on Thursday June 13, 2024 @02:32AM (#40666571)

This is absolutely the case, TPUs scale very well: https://github.com/google/maxtext .

[ Parent | on HN ]

Re:How Meta trains large language models at scale by pama on Thursday June 13, 2024 @02:53AM (#40666728)

The repo mentiones a Karpathy tweet from Jan 2023. Andrej has recently created llm.c and the same model trained about 32x faster on the same NVidia hardware mentioned in the tweet. I dont think the perfomance estimate that the repo used (based on that early tweet) was accurate for the performance of the NVidia hardware itself.

[ Parent | on HN ]

Re:How Meta trains large language models at scale by fbdab103 on Wednesday June 12, 2024 @11:24PM (#40665627)

Time is money. You might be a lab with long queues to train, leaving expensive staff twiddling their thumbs.

[ Parent | on HN ]