News for hackers. Stuff that matters.
Re:21.2× faster than llama.cpp? plus 40% memory usage reduction by russianGuy83829 on Wednesday June 12, 2024 @06:32PM (#40663922)
It seems like this can’t run all models, and needs custom ones trained from scratch: “ We introduce two new models: TurboSparse-Mistral-7B and TurboSparse-Mixtral-47B. These models are sparsified versions of Mistral and Mixtral […]. Notbly, our models are trained with just 150B tokens within just 0.1M dollars”.

It remains to be seen how good these custom models are.

Re:21.2× faster than llama.cpp? plus 40% memory usage reduction by TOMDM on Wednesday June 12, 2024 @06:41PM (#40663986)
Paper for the sparcified mixtral models

https://arxiv.org/abs/2406.05955

Re:21.2× faster than llama.cpp? plus 40% memory usage reduction by cpldcpu on Wednesday June 12, 2024 @08:14PM (#40664578)
This seems very similar to the other paper.
Re:21.2× faster than llama.cpp? plus 40% memory usage reduction by helloericsf on Wednesday June 12, 2024 @06:43PM (#40664009)
Agreed. custom models could be a hit or a miss.
Re:21.2× faster than llama.cpp? plus 40% memory usage reduction by cpldcpu on Wednesday June 12, 2024 @09:17PM (#40664944)
It's just continued pretraining to "heal" the damage caused by switching the activation functions and enforcing sparsity.

Apparently they managed to recover original performance on standardized tests after continuing pretraining with the 150B tokens. There may be some more specialized knowledge lost that was not covered by their dataset.