News for hackers. Stuff that matters.
Re:Lossless Acceleration of LLM via Adaptive N-Gram Parallel... by tripplyons on Sunday April 21, 2024 @09:15PM (#40110696)
The HuggingFace transformers library already has support for a similar method called prompt lookup decoding that uses the existing context to generate an ngram model: https://github.com/huggingface/transformers/issues/27722

I don't think it would be that hard to switch it out for a pretrained ngram model.

There are no comments on this comment yet.