Every story and comment from the Hacker News API, in a layout borrowed from 2001.
A comment on Running GPT-OSS-120B at 500 tokens per second on Nvidia GPUs. Parent comment
So your draft model can decode N new tokens, then the real model does one inference pass to score the N new drafted tokens.
Prefill is computation bound whereas decode is bandwidth bound, so in practice doing one prefill over N tokens is cheaper than doing N decode passes.
There are no comments on this comment yet.
So your draft model can decode N new tokens, then the real model does one inference pass to score the N new drafted tokens.
Prefill is computation bound whereas decode is bandwidth bound, so in practice doing one prefill over N tokens is cheaper than doing N decode passes.