For me it completely replaced strong models such as Mixtral-8x7B and DeepSeek-Coder-Instruct-33B.
1. https://www.reddit.com/r/LocalLLaMA/comments/1cst400/result_...
What, specifically, are you asking of these LLMs? "creative tasks" can be anything from programming to cooking recipes, so a tiny bit more specificality would be appreciated :)
I am too dumb for all of this ML stuff. Can you explain what exactly that means & why it's surprising?
More training leads to better compression. Trimming precision off the integers reveals that there was more compression that could be had.
Quantize each block or vector as a unit, bound to some metric instead of simply quantizing the whole thing all at once.
It's an interesting topic https://old.reddit.com/r/LocalLLaMA/comments/1ba55rj/overvie...
I did a rough calculation, and as the precision of the scalar weights decreases, the information content of the specific network permutation becomes a much higher percentage of its overall size.
Depending on the particular neural network architecture, there may be other symmetries beside the symmetric group that also represent compressible redundancies.
My intuition is that a model which is undertrained suffers less from quantization, because the training process has not utilized each weight to its full potential. One of the key findings with llama, and why it punches above its weight for its size, is that they trained it for longer on a much larger dataset then was "optimal" according to the literature up to that point.
Putting two and two together, it seems that:
small model, lots of data, long training > large model + quantization
That basically, quantization is a lossy shortcut to the tail of training long. Amount and quality of data is, as always, the most important part about all of this.