Re:Speculative Decoding in vLLM on AMD GPUs
by nicce on Monday September 07, 2026 @11:09AM
(#49599294)
2x RTX 3090 is not enough for proper use. ideal is 64GB+ so that you get proper cache and 256k context with good enough models. (e.g. Qwen 3.8 27B with MXFP4).
And that is tight already.
You would need 3x RTX 3090 - but then the PCIe bandwidth comes an issue if you really want tensor parallelism with three cards. 3x PCIe5 x16 is not cheap with direct CPU access.
So surprisingly, 2x r9700 starts be a nice deal.
> Also these 20-30t/s jumping to 150-200... Watch out for the massaged numbers coming from vendors.
Well, luckily these are not vendor numbers. Prefill also scales almost linearly with the amount of GPUs.
Re:Speculative Decoding in vLLM on AMD GPUs
by formerly_proven on Monday September 07, 2026 @12:21PM
(#49600035)
You don't need PCIe 5.0 x16 since RTX 30 are not PCIe 5.0 to begin with.
Re:Speculative Decoding in vLLM on AMD GPUs
by nicce on Monday September 07, 2026 @12:28PM
(#49600105)
Well, that makes them just even slower then
Re:Speculative Decoding in vLLM on AMD GPUs
by snovv_crash on Monday September 07, 2026 @01:21PM
(#49600658)
The whole point of dual 3090 over other cards is the nvlink support. At that point the pcie doesn't really matter.
Re:Speculative Decoding in vLLM on AMD GPUs
by nicce on Monday September 07, 2026 @02:26PM
(#49601354)
Oops
Re:Speculative Decoding in vLLM on AMD GPUs
by latentsea on Monday September 07, 2026 @09:59PM
(#49604876)
> 2x RTX 3090 is not enough for proper use
I don't see how. Even on 32GB you can run Q6_K_XL quant with MTP at 200k context k=q8_0, v=q5_1. So 48GB VRAM is good enough to run Q8 at long context. Also with tings like ninfer and it's various forks I'm seeing people get very good performance out of Qwen3.8 models on all sorts of NVIDIA cards.
Re:Speculative Decoding in vLLM on AMD GPUs
by nicce on Tuesday September 08, 2026 @05:12AM
(#49607698)
You also want prompt/prefix cache. Otherwise there is a lot of duplicate prefill processing if you fork conversation, have a different chat window or anything like that. It therefore also makes subagents much faster.
Only if AMD made a card like this with 48G+ I'd consider it.
Also these 20-30t/s jumping to 150-200... Watch out for the massaged numbers coming from vendors.
I believe Intel has claimed something like 1400tok/s (generation! Not prefill) of Qwen3.6-moe on Arc b70.
I was actually very interested in this so I checked the details. Turns out it was 200 simultaneous users running the same 1024 token prompt :D so all the experts got maximum parallelism.
How often are you going to run 200 parallel sessions with a tiny context and same prompt running at 7tok/s.
Based on how much my rtx3090 is getting on a single user (150tok/s) I'm estimating b70 to probably get less than that.
Sadly nvidia is king now.
Also, most of us already have nvidia cards and no inference software supports mixing let's say nvidia, Intel and amd cards in inference of one model.