iPhone 17 Pro Demonstrated Running a 400B LLM

Re:iPhone 17 Pro Demonstrated Running a 400B LLM by kgeist on Monday March 23, 2026 @04:16PM (#47494537)

>I suggest Q5 quantization at most from my experience. Q4 works on short responses but gets weird in longer conversations.

There are dynamic quants such as Unsloth which quantize only certain layers to Q4. Some layers are more sensitive to quantization than others. Smaller models are more sensitive to quantization than the larger ones. There are also different quantization algorithms, with different levels of degradation. So I think it's somewhat wrong to put "Q4" under one umbrella. It all depends.

[ Parent | on HN ]

Re:iPhone 17 Pro Demonstrated Running a 400B LLM by Aurornis on Monday March 23, 2026 @04:24PM (#47494622)

I should clarify that I'm referring generically to the types of quantizations used in local LLM inference, including those from Unsloth.

Nobody actually quantizes every layer to Q4 in a Q4 quant.

[ Parent | on HN ]