Qwen3.8-27B at 256K on a 24GB RTX PRO 4000 SFF (432 GB/s): 50 tok/s with MTP

Re:Qwen3.8-27B at 256K on a 24GB RTX PRO 4000 SFF (432 GB/s)... by petu on Monday August 17, 2026 @12:49PM (#49334007)

I guess it's due to testing on MoE. Different completions activate different experts, thus very little cache reuse and completions "steal" memory bandwidth from each other.

As I understand (useful) concurrency for MoE requires very large batches, where about every expert gets activated per pass.

With dense Qwen 27B on 3090/llama.cpp I get:

  - no MTP: 1x42, 2x33, 3x24, 4x19 t/s  
  - MTP: 1x50, 2x30, 3x33, 4x30 t/s

[ Parent | on HN ]

Re:Qwen3.8-27B at 256K on a 24GB RTX PRO 4000 SFF (432 GB/s)... by jermaustin1 on Monday August 17, 2026 @01:53PM (#49334970)

That is interesting. I'll have to test that theory out today.

[ Parent | on HN ]