joliu

It does run inference, but on the batch of tokens that were drafted, akin to the prefill phase.

So your draft model can decode N new tokens, then the real model does one inference pass to score the N new drafted tokens.

Prefill is computation bound whereas decode is bandwidth bound, so in practice doing one prefill over N tokens is cheaper than doing N decode passes.

Shape Rotator. That is the meme.

Doesn't a DSLR work by moving the mirror out of the way of the sensor when taking an image?

That may be true but as the popularity of Netflix exclusive content grows so does Netflix's ability to display ads without shedding as much of its user base.

Ruby's fastest time for pidigits was 3.14, haha.