News for hackers. Stuff that matters.
Re:Inkling 2 Mini by AndrewKemendo on Friday November 20, 2026 @04:18PM (#50582266)
Please reply when you have tested it I’m curious.

Also do you know what the best local setup is?

Re:Inkling 2 Mini by rsolva on Friday November 20, 2026 @04:27PM (#50582419)
For 7900XTX you mean? I run Fedora Silverblue 44 at the moment and could share my current containerized setup with DaisyRose 27B.
Re:Inkling 2 Mini by jakswa on Friday November 20, 2026 @04:30PM (#50582454)
I had astra patch in both required llama.cpp PRs and it's passing a few tests locally on my 7900XTX so far, but ~500 tok/s prefill and ~26 tok/s decode are sobering: The new three-time-looping might be cutting speeds. Maybe there's something here still though, maybe it needs less reasoning output for results to be usable/good? More testing needed!

edit: found some oudated mesa/radv/shaderc/glslang packages to update, and up to 34 tok/s now. MORE MORE

Re:Inkling 2 Mini by rsolva on Friday November 20, 2026 @04:35PM (#50582531)
Hm, that seems strangely slow. Qwen3.8-27B/DaisyRose-27B Q4 currently run with ~66 t/s and ~800 t/s prefill with 3 slots and about 190k kv cache. It performs really well so far.
Re:Inkling 2 Mini by jakswa on Friday November 20, 2026 @04:37PM (#50582563)
Yeah I've had 27B (dense model) up to similar speeds on this setup as well. I'll hold off to compare my speeds to others, maybe it's too early or it's particular to my setup.
Re:Inkling 2 Mini by enlyth on Friday November 20, 2026 @04:53PM (#50582795)
I'm getting 700 tok/s prefill and 52tok/s generation on a 5090 RTX, not bad
Re:Inkling 2 Mini by jakswa on Friday November 20, 2026 @05:08PM (#50582994)
5090 RTX is like 1800GB/s? I bet you were seeing much higher speeds on Qwen 3.8 27B right?
Re:Inkling 2 Mini by shadowpho on Friday November 20, 2026 @05:25PM (#50583203)
That seems slow, should be much much faster hmmm
Re:Inkling 2 Mini by baoooooooooooo on Friday November 20, 2026 @10:15PM (#50585549)
I'm always on the lookout for reports from my 3090 friends

Looking a bit oof on reddit rn, I'm seeing reports of 10 tok/s via llama.cpp :|

https://www.reddit.com/r/LocalLLaMA/comments/1y1g9cr/inkling...

Edit: Hopeful that this gets the ninfer treatment, if the benchmarks are to be believed it feels like there is probably both amazing speed and real performance to be unlocked by the community here on the x090 cards