News for hackers. Stuff that matters.
Re:Better and Faster Large Language Models via Multi-Token P... by cgearhart on Wednesday May 01, 2024 @11:17AM (#40224498)
What you’ve described is basically the problem with greedy sampling in the decoder. Many other local optimization sampling strategies exist (e.g., beam search) and there’s been a lot of work on more global sampling (e.g., speculative decoding).

There are no comments on this comment yet.