News for hackers. Stuff that matters.
Re:You can now train a 70B language model at home by sqreept on Friday March 08, 2024 @03:35AM (#39639100)
M1, M2, M3 still have very low number of GPU cores. Apple should release some better hardware to take advantage of their recently released MLX library.
Re:You can now train a 70B language model at home by sbinnee on Friday March 08, 2024 @07:24AM (#39640418)
At this moment it looks clear to me that Apple won’t go that way. It’s enough for them to focus on inference and actual application not the heavy training part. They have been probably training models on a cluster with non Apple silicon and make them available for their chips only for inference.
Re:You can now train a 70B language model at home by ttul on Friday March 08, 2024 @10:36AM (#39642009)
Not to mention entirely outsourcing training workloads to specialist firms. Apple does a lot of secretive outsourcing of things you might think they would or should do in-house. This contrasts with Google and Meta who seem to like keeping everything in-house.
Re:You can now train a 70B language model at home by kergonath on Friday March 08, 2024 @10:41AM (#39642074)
It’s true that their GPUs are slower than Nvidia’s. But keep in mind that cores are really different and cannot be compared across architectures. You want more Gflops, not necessarily more cores.
Re:You can now train a 70B language model at home by int_19h on Friday March 08, 2024 @02:38PM (#39645334)
They do, but for inference at least, it's memory bandwidth that is the primary limiting factor for home LLMs right now, not raw compute.
Re:You can now train a 70B language model at home by sroussey on Saturday March 09, 2024 @01:20PM (#39653661)
Wonder if the apple silicon ultra series will start using HBM3(e) on desktop in the future.