News for hackers. Stuff that matters.
Re:Inkling 2 Mini by meatmanek on Friday November 20, 2026 @07:57PM (#50584615)
I'm getting about 18 on the same hardware using the unsloth build of llama.cpp; I guess I should try upstream now that support is merged.

Theoretically with 400GB/s memory bandwidth and effectively 5.4B active parameters (accounting for the looping), token generation should be closer to 74 tokens/second. 18tps is close to what you'd expect for a dense model this size (22GB for UD-Q8_K_XL). Hopefully over the next few days, someone clever figures out what's going on and optimizes this.

Re:Inkling 2 Mini by dofm on Friday November 20, 2026 @09:33PM (#50585324)
> Theoretically with 400GB/s memory bandwidth

Well it would be theoretical, alas — the M1 Max can't actually saturate the bus with the GPU.

I have seen varying suggestions that it might in practice between 320GB/s and 390GB/s; for the CPU it is much much lower (only 243 GB/s).

I am seeing 16-20 tokens per second generation in Unsloth Desktop.