News for hackers. Stuff that matters.
Re:Lossless Acceleration of LLM via Adaptive N-Gram Parallel... by mirekrusin on Monday April 22, 2024 @12:45AM (#40111470)
They should support another layer of pre-generation that is controlled on client side/through grammar instead of n-gram markov chain / small network.

Especially when it comes to programming languages or data formats like json - a lot of compute is spent on ie. silly whitespaces.

There are no comments on this comment yet.