News for hackers. Stuff that matters.
Re:Show HN: Speeding up LLM inference 2x times (possibly) by kolinko on Friday April 19, 2024 @01:42AM (#40083757)
Yeah, more tests are needed. I got some feedback on using KL instead of the token similarity - initial tests seem to show that it is workable (compared to Q8), but not awesomely amazing - will be working on that next week and publishing.

As for treating effort+Mistral as a separate model - I wouldn't do that comparison. The model stays the same, all the weights from it are still being used, just not all of the time - we don't really lose information from the source model.

There are no comments on this comment yet.