News for hackers. Stuff that matters.

A comment on Qwen3.8-27B at 256K on a 24GB RTX PRO 4000 SFF (432 GB/s): 50 tok/s with MTP. Parent comment

Re:Qwen3.8-27B at 256K on a 24GB RTX PRO 4000 SFF (432 GB/s)... by dannyw on Monday August 17, 2026 @12:14PM (#49333413)
Your configuration is broken or wrong. What are you using? Hopefully not llama.cpp?

I’ve sweeped concurrency across many models and many different kinds of hardware, and the only times I saw similar results to you were when I didn’t configure it correctly.

[ Parent | on HN ]
Re:Qwen3.8-27B at 256K on a 24GB RTX PRO 4000 SFF (432 GB/s)... by mhitza on Monday August 17, 2026 @01:00PM (#49334178)
What do you use instead of llama.cpp? With vllm for example most models don't seem to be supported out of the box.
[ Parent | on HN ]