News for hackers. Stuff that matters.
Re:Better and Faster Large Language Models via Multi-Token P... by henderson98 on Friday May 03, 2024 @06:58PM (#40253271)
Hmm, I may have misundertood this, but isnt this just beam search but multiple times(* possibly). Also usually the search is over the discrete token space directly, I am not sure if there are continous surrogates which translate the discrete inference problem(combinatorial) to a continous one, which fit better with your local minima, and perturbation terminology. Although I am uncertain about the utility of beam search run multiple times, I am keen to research literature casting the inference search problem as a continous one.

*This might be just a detail/semantics, but for example, for a 3 word context, your starting points may look like "[some token][empty][empty]". Here your procedure simply reduces to a single run of beam search, not multiple, since beam search optimises locally for every turn, producing n different "perturbations" every turn. Let me know if I misunderstood you on this.

But inference combinatorial optimisation as a continous surrogate(which sounds like what you are conveying, and will naturally result in starting points as sentences and not just incomplete words/sequences) is something I never considered. There must be some literature around on this....lets see.

There are no comments on this comment yet.