Re:Claude Opus 4.8
by amluto on Thursday May 28, 2026 @03:05PM
(#48313854)
How useful is speculative decoding in a batched setting where you get paid for throughput (aggregated across users) and you mostly don’t get paid for latency or single-session throughput?
[ Parent
| on HN ]
Re:Claude Opus 4.8
by onlyrealcuzzo on Thursday May 28, 2026 @03:07PM
(#48313894)
It's useful at the local level, where there will be SOTA models developed...
Local models are moving towards batched inference too, if only for non-interactive use. An early experimental patchset for DS4 (running DeepSeek V4 Flash) seems to show 2x aggregate tok/s decode when processing 8 streams concurrently, and more than 3x when processing as many as 32 streams concurrently. Note that decode (which is not helped significantly) then becomes a much larger portion of total wall-clock time, so the overall gain is lower. MTP will still be highly valuable for interactive use of course.