News for hackers. Stuff that matters.
Re:M5 Ultra Mac Studio Review: The Dream Mac for Local AI Ag... by redox99 on Monday September 21, 2026 @11:05AM (#49788252)
A dense 27B doesn't really make sense for the Mac. A MoE makes way more sense when you have modest bandwidth but lots of memory.
Re:M5 Ultra Mac Studio Review: The Dream Mac for Local AI Ag... by peri-cl on Monday September 21, 2026 @11:12AM (#49788354)
They do MoE. They benchmarked GLM 5.3-flash (320B / 18B), and Qwen 3.8-flash-next (125B / 6B). The dense Qwen is only focused (I assume) because it's about the only thing that fits on a 5090, that they can compare the two heads on.
Re:M5 Ultra Mac Studio Review: The Dream Mac for Local AI Ag... by tcdent on Monday September 21, 2026 @01:09PM (#49790129)
A dense model (up to the amount of memory available) actually does make the most sense on unified memory architectures. But when you hit the limit of what you can hold in memory, you reach the limitation of the platform.

Whereas a hybrid architecture with distinct DRAM and VRAM with sparse MoE, you can leverage two different bit rates depending on the actual need for constant access to common layers versus sparse access to infrequent layers and arbitrage the difference in cost for each of those in distinct classes of hardware.

Re:M5 Ultra Mac Studio Review: The Dream Mac for Local AI Ag... by nojs on Monday September 21, 2026 @06:36PM (#49794382)
> A dense model (up to the amount of memory available) actually does make the most sense on unified memory architectures

Inference time is going to be dominated by the low memory bandwidth on these Macs, so a dense model will suffer most. It’s more of an opportunity for large MoE models with a low number of active experts since you can keep all experts in VRAM but not pay the bandwidth cost until they are used.

> you can leverage two different bit rates depending on the actual need for constant access to common layers versus sparse access to infrequent layers

This is an interesting direction that I expect to see more of. But for most models currently you need basically all experts loaded since they are chosen per token.

Apple seems to be researching longer horizon expert caching, where they keep experts swapped in for longer runs of tokens [1]. Other labs are offloading ngram caches but not sure if they’re pursuing anything like this?

1. https://machinelearning.apple.com/research/introducing-third...

Re:M5 Ultra Mac Studio Review: The Dream Mac for Local AI Ag... by thejazzman on Monday September 21, 2026 @10:38PM (#49796214)
1.2TB/s is already considered slow? Things are moving quickly!
Re:M5 Ultra Mac Studio Review: The Dream Mac for Local AI Ag... by skohan on Monday September 21, 2026 @04:46PM (#49793148)
1.2 T/s is not that modest is it? That's very close to an RTX pro 5000