News for hackers. Stuff that matters.

A comment on Qwen3.8-27B at 256K on a 24GB RTX PRO 4000 SFF (432 GB/s): 50 tok/s with MTP. Parent comment

Re:Qwen3.8-27B at 256K on a 24GB RTX PRO 4000 SFF (432 GB/s)... by pich on Monday August 17, 2026 @12:34PM (#49333781)
A 3090 has 936 GB/s of bandwidth vs 432 GB/s on the 70W RTX PRO 4000 SFF, so 70 t/s there is not surprising. The interesting constraint here was fitting a workload-tuned 5.01 BPW quant + 256K + MTP into 24 GB while working with less than half the memory bandwidth
[ Parent | on HN ]

There are no comments on this comment yet.