henderson98
Re:Better and Faster Large Language Models via Multi-Token Prediction
on Friday May 03, 2024 @06:58PM (#40253271)
Hmm, I may have misundertood this, but isnt this just beam search but multiple times(* possibly). Also usually the search is over the discrete token space directly, I am not sure if there are continous surrogates which translate the discrete inference problem(combinatorial) to a continous one, which fit better with your local minima, and perturbation terminology. Although I am uncertain about the utility of beam search run multiple times, I am keen to research literature casting the inference search problem as a continous one.
*This might be just a detail/semantics, but for example, for a 3 word context, your starting points may look like "[some token][empty][empty]". Here your procedure simply reduces to a single run of beam search, not multiple, since beam search optimises locally for every turn, producing n different "perturbations" every turn. Let me know if I misunderstood you on this.
But inference combinatorial optimisation as a continous surrogate(which sounds like what you are conveying, and will naturally result in starting points as sentences and not just incomplete words/sequences) is something I never considered. There must be some literature around on this....lets see.
Re:Better and Faster Large Language Models via Multi-Token Prediction
on Friday May 03, 2024 @05:10AM (#40245667)
....uhh Isnt this what Beam search attempts to fix(approximate) over greedy decoding. You know greedy optimum vs global optimum. And most LM already use beam search. Because finding the true optimum is intractable even over modest lengths.