News for hackers. Stuff that matters.
Re:Running GPT-OSS-120B at 500 tokens per second on Nvidia GPUs by joliu on Thursday August 07, 2025 @01:17AM (#44820854)
It does run inference, but on the batch of tokens that were drafted, akin to the prefill phase.

So your draft model can decode N new tokens, then the real model does one inference pass to score the N new drafted tokens.

Prefill is computation bound whereas decode is bandwidth bound, so in practice doing one prefill over N tokens is cheaper than doing N decode passes.

There are no comments on this comment yet.