News for hackers. Stuff that matters.

A comment on Apple unveils M3, M3 Pro, and M3 Max, the most advanced chips for a PC. Parent comment

Re:Apple unveils M3, M3 Pro, and M3 Max, the most advanced c... by sunpazed on Tuesday October 31, 2023 @04:54AM (#38081904)
Well, Metal can only allocate a smaller portion of “VRAM” to the GPU — about 70% or so, see; https://developer.apple.com/videos/play/tech-talks/10580

If you want to run larger models, then CPU inference is your only choice.

[ Parent | on HN ]
Re:Apple unveils M3, M3 Pro, and M3 Max, the most advanced c... by __loam on Tuesday October 31, 2023 @01:06PM (#38088014)
Aren't these things supposed to have cores dedicated to ml?
[ Parent | on HN ]
Re:Apple unveils M3, M3 Pro, and M3 Max, the most advanced c... by brucethemoose2 on Wednesday November 01, 2023 @11:57AM (#38100100)
They are not as fast as the GPU (but much lower power).

Also, not many implementations can even use it.

[ Parent | on HN ]
Re:Apple unveils M3, M3 Pro, and M3 Max, the most advanced c... by azinman2 on Wednesday November 01, 2023 @12:27PM (#38100659)
You’re thinking of the neural engine. I’m not sure that llama.cpp makes use of this. They’d have to turn it into a CoreML model to do so.
[ Parent | on HN ]