News for hackers. Stuff that matters.
Re:Lossless Acceleration of LLM via Adaptive N-Gram Parallel... by MyFirstSass on Sunday April 21, 2024 @03:38PM (#40108563)
If this is "plug and play" can it be added to say llama.cpp and give ~3.67 speedup to existing models or is there some complication?
Re:Lossless Acceleration of LLM via Adaptive N-Gram Parallel... by jncraton on Sunday April 21, 2024 @04:02PM (#40108740)
The speedup would not be that high in practice for folks already using speculative decoding[1]. ANPD is similar but uses a simpler and faster drafting approach. These two enhancements can't be meaningfully stacked. Here's how the paper describes it:

> ANPD dynamically generates draft outputs via an adaptive N-gram module using real-time statistics, after which the drafts are verified by the LLM. This characteristic is exactly the difference between ANPD and the previous speculative decoding methods.

ANPD does provide a more general-purpose solution to drafting that does not require training, loading, and running draft LLMs.

[1] https://github.com/ggerganov/llama.cpp/pull/2926

Re:Lossless Acceleration of LLM via Adaptive N-Gram Parallel... by MacsHeadroom on Sunday April 21, 2024 @10:24PM (#40110972)
Who is already using speculative decoding? I haven't seen anything about it in the llama.cpp or ollama docs.
Re:Lossless Acceleration of LLM via Adaptive N-Gram Parallel... by eshoyuan on Sunday April 21, 2024 @10:59PM (#40111102)
Re:Lossless Acceleration of LLM via Adaptive N-Gram Parallel... by tripplyons on Sunday April 21, 2024 @09:15PM (#40110696)
The HuggingFace transformers library already has support for a similar method called prompt lookup decoding that uses the existing context to generate an ngram model: https://github.com/huggingface/transformers/issues/27722

I don't think it would be that hard to switch it out for a pretrained ngram model.