News for hackers. Stuff that matters.
Re:Better and Faster Large Language Models via Multi-Token P... by wongarsu on Wednesday May 01, 2024 @05:27AM (#40221144)
They just throw the predictions for +1 and +2 away, and only generate them for more efficient training.

The abstract doesn't make that clear, but from the description of figure 1: "During inference, we employ only the next-token output head. Optionally, the other three heads may be used to speed-up inference time"

Maybe you can use all three heads if you take the top prediction from all of them, but that prevents you from doing any of the common sampling strategies. I'm not sure how many people actually run an LLM with temperature 0 outside of benchmarks, unless they do something even better than applying a temperature

Re:Better and Faster Large Language Models via Multi-Token P... by faabian on Wednesday May 01, 2024 @06:02AM (#40221329)
Exactly, but there is also a rejection sampling based method for speculative sampling: https://arxiv.org/abs/2302.01318
Re:Better and Faster Large Language Models via Multi-Token P... by Havoc on Wednesday May 01, 2024 @06:40AM (#40221571)
Thanks for explaining