Qwen3.8-27B at 256K on a 24GB RTX PRO 4000 SFF (432 GB/s): 50 tok/s with MTP

Re:Qwen3.8-27B at 256K on a 24GB RTX PRO 4000 SFF (432 GB/s)... by Foobar8568 on Monday August 17, 2026 @02:08PM (#49335202)

Looks like slope benchmarks and results, and as usually, people are mixing MTP numbers with non MTP numbers. Or just 100 token input benchmarks. Or just failed ones as actual measures.

https://github.com/Neroued/ninfer/blob/master/docs/performan...

Category MTP3 stochastic sampler DFlash stochastic sampler DFlash greedy Code 1/15 natural stops; 0/15 prompt-complete 2/15 natural stops; 0/15 prompt-complete 0/15 natural stops Story 9/15 natural stops; the nine Chinese outputs pass requested division and minimum length 8/15 natural stops; the eight Chinese outputs pass requested division and minimum length 10/15 natural stops; five Chinese dialogue outputs are under length Translation 15/15 natural stops; 15/15 pass structural checks 15/15 natural stops; 15/15 pass structural checks 15/15 natural stops; 15/15 pass structural checks Structured 0/15 satisfy the requested complete record/script contract 0/15 satisfy the requested complete record/script contract 0/15 satisfy the requested complete record/script contract

And on my "own" "quick" benchmark, it's slower than vllm.

[ Parent | on HN ]

Re:Qwen3.8-27B at 256K on a 24GB RTX PRO 4000 SFF (432 GB/s)... by iv42 on Monday August 17, 2026 @03:02PM (#49335987)

I don't have a 5090, so I can't really comment, but here's the relevant reddit thread from today where they report the numbers (including ninfer ones), and where you can make your case: https://www.reddit.com/r/LocalLLaMA/comments/1vqjeub/how_man...

[ Parent | on HN ]