News for hackers. Stuff that matters.
Re:Run LLMs on Apple Neural Engine (ANE) by behnamoh on Saturday May 03, 2025 @03:22PM (#43881413)
> LLM performance is twice as fast as RTX 5090

your tests are wrong. you used MLX for Mac Studio (optimized for Apple Silicon) but you didn't use vLLM for 5090. There's no way a machine with half the bandwidth of 5090 delivers twice as fast tok/s.

Re:Run LLMs on Apple Neural Engine (ANE) by voidspark on Saturday May 03, 2025 @03:33PM (#43881490)
Yeah that's probably wrong. But the M3 Ultra is good enough for local inferencing, in any case.
Re:Run LLMs on Apple Neural Engine (ANE) by seanmcdirmid on Saturday May 03, 2025 @03:23PM (#43881418)
Unless it’s a large model that doesn’t fit in the 5090, bust that’s no longer a $4k macstudio I think.
Re:Run LLMs on Apple Neural Engine (ANE) by voidspark on Saturday May 03, 2025 @03:58PM (#43881728)
$4k will get you a 96 GB Mac Studio with M3 Ultra (819 GB/sec).

That's 3x the RAM of the 5090.

Re:Run LLMs on Apple Neural Engine (ANE) by mdp2021 on Saturday May 03, 2025 @07:47PM (#43883295)
> That's 3x the RAM of the 5090

And a bit less than half the bandwidth (saying for completeness).

Re:Run LLMs on Apple Neural Engine (ANE) by behnamoh on Saturday May 03, 2025 @03:30PM (#43881469)
that's orthogonal to the speed discussion.

also, the GP was mostly testing models that fit in both 5090 and Mac Studio.