Re:Running GPT-OSS-120B at 500 tokens per second on Nvidia GPUs
by cristoperb on Thursday August 07, 2025 @01:35AM
(#44820945)
My simplified understanding: The target model can validate the draft tokens all at once, in a single forward pass. The output of that forward pass is a list of probabilities for each draft token which are compared to the probabilities produced by the draft model. If the target model's probabilities are the same or greater than the draft model, the tokens are accepted. Worst case none of the draft tokens are accepted and instead the target model selects the single next token as usual.
[ Parent
| on HN ]
There are no comments on this comment yet.