News for hackers. Stuff that matters.

A comment on iPhone 17 Pro Demonstrated Running a 400B LLM. Parent comment

Re:iPhone 17 Pro Demonstrated Running a 400B LLM by snovv_crash on Monday March 23, 2026 @11:57AM (#47491255)
The real improvement will be when the software engineers get into the training loop. Then we can have MoE that use cache-friendly expert utilisation and maybe even learned prefetching for what the next experts will be.
[ Parent | on HN ]
Re:iPhone 17 Pro Demonstrated Running a 400B LLM by zozbot234 on Monday March 23, 2026 @12:29PM (#47491710)
> maybe even learned prefetching for what the next experts will be

Experts are predicted by layer and the individual layer reads are quite small, so this is not really feasible. There's just not enough information to guide a prefetch.

[ Parent | on HN ]
Re:iPhone 17 Pro Demonstrated Running a 400B LLM by yorwba on Monday March 23, 2026 @01:26PM (#47492489)
It's feasible to put the expert routing logic in a previous layer. People have done it: https://arxiv.org/abs/2507.20984
[ Parent | on HN ]
Re:iPhone 17 Pro Demonstrated Running a 400B LLM by snovv_crash on Monday March 23, 2026 @12:34PM (#47491785)
Manually no. It would have to be learned, and making the expert selection predictable would need to be a training metric to minimize.
[ Parent | on HN ]
Re:iPhone 17 Pro Demonstrated Running a 400B LLM by zozbot234 on Monday March 23, 2026 @12:40PM (#47491851)
Making the expert selection more predictable also means making it less effective. There's no real free lunch.
[ Parent | on HN ]