Re:Mesh LLM: distributed AI computing on iroh
by zdw on Saturday July 11, 2026 @08:44PM
(#48877253)
What hardware (CPU/GPU/memory) and network was used for this? What quantization for GLM 5.2? How much tuning of the split was needed?
[ Parent
| on HN ]
Re:Mesh LLM: distributed AI computing on iroh
by i386 on Saturday July 11, 2026 @09:12PM
(#48877390)
The lab features two Mac Studios: an Apple M3 Ultra (32 CPU cores, 80 GPU cores, 256 GB unified memory) and an Apple M1 Ultra (20 CPU cores, 48 GPU cores, 128 GB unified memory), both connected via 1Gbit Ethernet.
We use a customized Q2 quantization that preserves sensitive tensors at Q8.
To reduce compute time per layer, we are developing a custom GLM DSA Metal graph.
While we are not yet approaching MTP, we plan to port our existing MTP implementations from versions 4.7 and 5.1 to 5.2.
Since GLM's MTP acceptance rate is very high for a single predicted token, we are exploring token prediction techniques to widen the predicted tokens and utilize parallelism for verification.
Re:Mesh LLM: distributed AI computing on iroh
by wlesieutre on Saturday July 11, 2026 @09:37PM
(#48877510)
Equivalent M3 machines no longer for sale from Apple (only up to 96 GB) but can be had on eBay for around $14,000 each
Re:Mesh LLM: distributed AI computing on iroh
by drawnwren on Sunday July 12, 2026 @02:29AM
(#48878855)
It's notable that they're so valuable because they feature 800Gbps of memory bandwidth. About twice what's available on the top end of M5, and exactly what makes llm inference fast.
Re:Mesh LLM: distributed AI computing on iroh
by embedding-shape on Sunday July 12, 2026 @05:17AM
(#48879620)
> because they feature 800Gbps of memory bandwidth. About twice what's available on the top end of M5
Ouch, about half of the memory bandwidth of a dedicated GPU though :/ Running LLMs on Apple hardware still doesn't make any sense to me.