News for hackers. Stuff that matters.

A comment on Run LLMs on Apple Neural Engine (ANE).

Re:Run LLMs on Apple Neural Engine (ANE) by jvstokes on Saturday May 03, 2025 @03:03PM (#43881303)
But coreml utilizes ANE, right? Is there some bottleneck in coreml that requires lower level access?
[ Parent | on HN ]
Re:Run LLMs on Apple Neural Engine (ANE) by anemll on Sunday May 04, 2025 @12:25AM (#43884485)
Memory bandwidth is the main bottleneck. It got better with M3/M4. ANE is really fast in FLOPS but low in memory bandwidth.
[ Parent | on HN ]