News for hackers. Stuff that matters.
Re:Lossless Acceleration of LLM via Adaptive N-Gram Parallel... by johntb86 on Sunday April 21, 2024 @03:48PM (#40108632)
They mention previous work on speculative decoding using similar techniques, but "ANPD dynamically generates draft outputs via an adaptive N-gram module using real-time statistics, after which the drafts are verified by the LLM. This characteristic is exactly the difference between ANPD and the previous speculative decoding methods."

There are no comments on this comment yet.