News for hackers. Stuff that matters.
Re:Lossless Acceleration of LLM via Adaptive N-Gram Parallel... by nsagent on Sunday April 21, 2024 @03:49PM (#40108647)
How does this differ from the 2018 NeurIPS paper, Blockwise Parallel Decoding for Deep Autoregressive Models?

https://arxiv.org/abs/1811.03115

Re:Lossless Acceleration of LLM via Adaptive N-Gram Parallel... by tripplyons on Sunday April 21, 2024 @09:11PM (#40110679)
They use a separate ngram model to generate the proposed sequence instead of extra heads on top of the main model. The process of verifying the proposed sequence appears to be the same.