Re:Running GPT-OSS-120B at 500 tokens per second on Nvidia GPUs
by sanxiyn on Thursday August 07, 2025 @02:49AM
(#44821359)
Yes, models give likelihoods you can compare against. No, you can't do that without drafting, because likelihood of token N+2 depends on token N+1. That is, you get P(is, The capital of France) and P(Berlin, The capital of France is), but for the later you need to give "is" as input, you can't do P(Berlin, The Capital of France _).
[ Parent
| on HN ]
There are no comments on this comment yet.