Re:A 10 year old Xeon is all you need (for 26B-A4B MTP Draft...
by dvdkon on Monday June 01, 2026 @08:23AM
(#48355890)
As far as I know, speculative decoding still verifies that the proposed tokens are what the "big" model would generate, it just uses the guesses to make that process faster. Setting the probability threshold too low then shouldn't affect correctness, just speed (time will be wasted verifying bad guesses).
Re:A 10 year old Xeon is all you need (for 26B-A4B MTP Draft...
by lreeves on Monday June 01, 2026 @08:26AM
(#48355916)
But won't setting it to accept 100% of the proposed tokens will skip the verification?
Re:A 10 year old Xeon is all you need (for 26B-A4B MTP Draft...
by naasking on Monday June 01, 2026 @08:26AM
(#48355914)
It depends on the type of MTP. If you're using two models, draft + full, then arguably yes, the larger model isn't providing much benefit if you really are seeing 100% acceptance rates. There are other forms of speculative decoding that work within the larger model by itself though, eg. Qwen has additional speculative decoding attention heads, so there is no secondary drafting model.
Also I feel like everyone leaves off prompt processing/prefill speeds in these articles. If you are using a very small prompt and asking for mostly generated tokens, sure but I'd love to know the time-to-response of asking for an analysis of an image or a few hundred lines of code.