News for hackers. Stuff that matters.
Re:Mistral 7B, the most powerful language model for its size... by kirill5pol on Wednesday September 27, 2023 @01:33PM (#37678136)
One big one is speculative decoding for larger models, the more “accurate”* the smaller model, the more the speed up on the bigger model

* as in matches the token that the larger model would output

Re:Mistral 7B, the most powerful language model for its size... by sroussey on Wednesday September 27, 2023 @01:38PM (#37678208)
It would be better if they also had a 70B model for this.

They would need the same vocabulary, etc. What else?