News for hackers. Stuff that matters.
Re:Apple Introduces M2 Ultra by sillysaurusx on Monday June 05, 2023 @06:25PM (#36204943)
There’s no difference between inference and training. Training is simply ~4x more expensive inference.

The submitted article also talks about training models.

Re:Apple Introduces M2 Ultra by bufo on Tuesday June 06, 2023 @10:03AM (#36212947)
There is a difference. We train with large batch sizes these days. The ANE silicon size is tiny and can't do the large matrix multiplications for big LLMs with or without a batch size higher than 1. Meaning that it cannot saturate the RAM bandwidth and that you're better using off the much bigger GPU on the Apple die.