Re:How Meta trains large language models at scale
by logicchains on Thursday June 13, 2024 @02:32AM
(#40666571)
This is absolutely the case, TPUs scale very well: https://github.com/google/maxtext .
[ Parent
| on HN ]
Re:How Meta trains large language models at scale
by pama on Thursday June 13, 2024 @02:53AM
(#40666728)
The repo mentiones a Karpathy tweet from Jan 2023. Andrej has recently created llm.c and the same model trained about 32x faster on the same NVidia hardware mentioned in the tweet. I dont think the perfomance estimate that the repo used (based on that early tweet) was accurate for the performance of the NVidia hardware itself.