Every story and comment from the Hacker News API, in a layout borrowed from 2001.
A comment on We got 207 tok/s with Qwen3.5-27B on an RTX 3090. Parent comment
This is interesting, doesn't greedy-only decoding slow down speculative decoding significantly?
In theory the probability of needing resampling (rejection) is (p_real-p_sample)+, which should be much smaller with non-greedy distribution
There are no comments on this comment yet.
This is interesting, doesn't greedy-only decoding slow down speculative decoding significantly?
In theory the probability of needing resampling (rejection) is (p_real-p_sample)+, which should be much smaller with non-greedy distribution