News for hackers. Stuff that matters.

A comment on Run LLMs on Apple Neural Engine (ANE). Parent comment

Re:Run LLMs on Apple Neural Engine (ANE) by anemll on Sunday May 04, 2025 @12:23AM (#43884470)
What hardware are you on? Most models are memory bandwidth limited. ANE was limited to 64GB/s prior to M3 Max or M4 pro. If you are on M1, GPU will be significantly faster for 3-8B models due to memory bandwidth rather then ANE capabilities.
[ Parent | on HN ]
Re:Run LLMs on Apple Neural Engine (ANE) by SparkyMcUnicorn on Sunday May 04, 2025 @12:36AM (#43884518)
M4 Max with 128GB of memory.
[ Parent | on HN ]
Re:Run LLMs on Apple Neural Engine (ANE) by anemll on Wednesday May 07, 2025 @12:14PM (#43917532)
M4 max should work at 120GB for ANE and 500+ for GPU. So GPU will be 3-4 times faster for anything over 1-3B. ANE is likely as fast for prefill due to higher FLOPs
[ Parent | on HN ]