Re:21.2× faster than llama.cpp? plus 40% memory usage reduction
by cpldcpu on Wednesday June 12, 2024 @09:17PM
(#40664944)
It's just continued pretraining to "heal" the damage caused by switching the activation functions and enforcing sparsity.
Apparently they managed to recover original performance on standardized tests after continuing pretraining with the 150B tokens. There may be some more specialized knowledge lost that was not covered by their dataset.
Apparently they managed to recover original performance on standardized tests after continuing pretraining with the 150B tokens. There may be some more specialized knowledge lost that was not covered by their dataset.