News for hackers. Stuff that matters.
Re:Cerebras Inference now 3x faster: Llama3.1-70B breaks 2,1... by bubblethink on Friday October 25, 2024 @02:43AM (#41942820)
Speculative decoding does not trade off accuracy. You reject the speculated tokens if the original model does not accept them, kind of like branch prediction. All these providers and third parties benchmark each other's solutions, so if there is a drop in accuracy, someone will report it. Their sequence length is 8k.

There are no comments on this comment yet.