Qwen3.8-27B at 256K on a 24GB RTX PRO 4000 SFF (432 GB/s): 50 tok/s with MTP

Re:Qwen3.8-27B at 256K on a 24GB RTX PRO 4000 SFF (432 GB/s)... by jermaustin1 on Monday August 17, 2026 @11:47AM (#49332929)

Because concurrency with a single "accessible" card quickly diminishes. I have dual 3090s, and on Qwen 3.6 35B A3B at 80k a single card with max concurrent set to 4 will get 70-80tps single request, 50-60 TOTAL tps with 2, 45-50 with 3, and around 40tps with all 4 going.

I know this doesn't exactly match your request, but I'm happy with my 3.6's performance. I've seen people claim they can get 120+tps on a 3090, but I'm not impatient when it comes to streaming text faster than I can read it.

[ Parent | on HN ]

Re:Qwen3.8-27B at 256K on a 24GB RTX PRO 4000 SFF (432 GB/s)... by dannyw on Monday August 17, 2026 @12:14PM (#49333413)

Your configuration is broken or wrong. What are you using? Hopefully not llama.cpp?

I’ve sweeped concurrency across many models and many different kinds of hardware, and the only times I saw similar results to you were when I didn’t configure it correctly.

[ Parent | on HN ]

Re:Qwen3.8-27B at 256K on a 24GB RTX PRO 4000 SFF (432 GB/s)... by mhitza on Monday August 17, 2026 @01:00PM (#49334178)

What do you use instead of llama.cpp? With vllm for example most models don't seem to be supported out of the box.

[ Parent | on HN ]

Re:Qwen3.8-27B at 256K on a 24GB RTX PRO 4000 SFF (432 GB/s)... by Tostino on Monday August 17, 2026 @11:52AM (#49333015)

You have something misconfigured then. Concurrency has never lowered my overall TPS. Also have dual 3090s. Generally use vllm though.

[ Parent | on HN ]

Re:Qwen3.8-27B at 256K on a 24GB RTX PRO 4000 SFF (432 GB/s)... by jermaustin1 on Monday August 17, 2026 @12:05PM (#49333230)

I've had some rough time getting LM-Studio properly configured for multi-card. It exists, but I feel like it is kind of buggy. I will disable a card and it will still load the model into it. Sometimes it will split the model even though there is loads of room available.

I might need to finally make the switch away from it, but it is so convenient, especially as a chat interface for system prompt experimentation.

[ Parent | on HN ]

Re:Qwen3.8-27B at 256K on a 24GB RTX PRO 4000 SFF (432 GB/s)... by pich on Monday August 17, 2026 @12:09PM (#49333299)

vLLM is probably the key difference there… its scheduler is built around batching/concurrency, while this setup is heavily optimized llama.cpp for single-stream latency

[ Parent | on HN ]

Re:Qwen3.8-27B at 256K on a 24GB RTX PRO 4000 SFF (432 GB/s)... by supermatt on Monday August 17, 2026 @12:22PM (#49333562)

I haven’t tried any larger models, but a 12B model on my Ampere A5000 gets around 4-5x the aggregate throughput with concurrency. I have the maximum context configured to 32k, but my actual requests are usually around 2-4k tokens.

No idea how that compares to running a larger model and context though.

[ Parent | on HN ]

Re:Qwen3.8-27B at 256K on a 24GB RTX PRO 4000 SFF (432 GB/s)... by petu on Monday August 17, 2026 @12:49PM (#49334007)

I guess it's due to testing on MoE. Different completions activate different experts, thus very little cache reuse and completions "steal" memory bandwidth from each other.

As I understand (useful) concurrency for MoE requires very large batches, where about every expert gets activated per pass.

With dense Qwen 27B on 3090/llama.cpp I get:

  - no MTP: 1x42, 2x33, 3x24, 4x19 t/s  
  - MTP: 1x50, 2x30, 3x33, 4x30 t/s

[ Parent | on HN ]

Re:Qwen3.8-27B at 256K on a 24GB RTX PRO 4000 SFF (432 GB/s)... by jermaustin1 on Monday August 17, 2026 @01:53PM (#49334970)

That is interesting. I'll have to test that theory out today.

[ Parent | on HN ]