What has changed in CPU cores in M3 chips?

Re:What has changed in CPU cores in M3 chips? by bernaferrari on Wednesday November 22, 2023 @07:05PM (#38387136)

even if that's true, the A series neural engine is much worse than the M series, I'll say it wrong, but it can only do 32bit inference (or something like that) where M series can do 64bit, so A series can run LLM but has a series of limitations that M series doesn't.

[ Parent | on HN ]

Re:What has changed in CPU cores in M3 chips? by icyfox on Wednesday November 22, 2023 @07:17PM (#38387307)

Practically speaking, most models today infer at 8bit or 16bit (sometimes, rarely 32). You don't see an empirical lift at more bits of precision. Size of the memory is far more important.

[ Parent | on HN ]

Re:What has changed in CPU cores in M3 chips? by OJFord on Wednesday November 22, 2023 @07:45PM (#38387602)

If we're talking about the results, is there any reason to think it should make a difference at all?

[ Parent | on HN ]

Re:What has changed in CPU cores in M3 chips? by icyfox on Thursday November 23, 2023 @10:48AM (#38394025)

Sometimes gradients are small but meaningful, if you constrain them to too few bits / degrees of freedom they'll be unable to backprop successfully. This can hamper training and therefore results quality.

You can also think about it as compounding errors - at any one weight index the bit values might not be too meaningful, but cascaded over a lot of tensor multiplications they will be.

[ Parent | on HN ]

Re:What has changed in CPU cores in M3 chips? by OJFord on Thursday November 23, 2023 @01:16PM (#38395731)

Oh I was thinking we were talking about the same calculations on different hardware.

[ Parent | on HN ]