News for hackers. Stuff that matters.
Re:The Ultra-Scale Playbook: Training LLMs on GPU Clusters by maxrmk on Wednesday February 19, 2025 @04:24PM (#43107902)
Ok this is way too long to read in one sitting, but it looks incredible? I've been looking for resources like this for """real""" model training at scale. The authors have worked on BLOOM (the first >100B parameter model), starcoder and fineweb so they actually know this space.

I wish it were not a single giant webpage, but I guess I can deal with that.

Re:The Ultra-Scale Playbook: Training LLMs on GPU Clusters by choppaface on Wednesday February 19, 2025 @11:28PM (#43111150)
Hugginface is mostly AWS, so these experiments might be done on an AWS-provided cluster? I wonder how many of the results are reproducible on "open market" clusters listed on e.g. https://gpulist.ai/
Re:The Ultra-Scale Playbook: Training LLMs on GPU Clusters by fragebogen on Friday February 21, 2025 @05:14AM (#43125922)
Kudos to Hugging face for going the extra mile on open sourcing knowledge. They could have "just" created a hub for model downloads, but the fact that they keep publishing top-tier tutorials, materials, experiments and code that benefits the whole community is such a blessing. IMO they're one of the most important players advancing the open source community effort. Keep it up!