Re:Lossless Acceleration of LLM via Adaptive N-Gram Parallel...
by MacsHeadroom on Sunday April 21, 2024 @10:24PM
(#40110972)
Who is already using speculative decoding? I haven't seen anything about it in the llama.cpp or ollama docs.
> ANPD dynamically generates draft outputs via an adaptive N-gram module using real-time statistics, after which the drafts are verified by the LLM. This characteristic is exactly the difference between ANPD and the previous speculative decoding methods.
ANPD does provide a more general-purpose solution to drafting that does not require training, loading, and running draft LLMs.
[1] https://github.com/ggerganov/llama.cpp/pull/2926