Re:GGUF, the Long Way Around
by rahimnathwani on Monday March 04, 2024 @07:28PM
(#39597955)
Why are you horrified?
In designing software, there's often a trade off between (i) generality / configurability, and (ii) performance.
llama.cpp is built for inference, not for training or model architecture research. It seems reasonable to optimize for performance, which is what ~100% of llama.cpp users care about.
Re:GGUF, the Long Way Around
by throwawaybbq1 on Wednesday March 06, 2024 @04:33AM
(#39614048)
GGUF files seems to be proliferating. I think some folks (like myself) make an incorrect assumption that the format has more portability/generalizability than it appears to have. Hence, the horror!
I'm also a bit confused by the quantization aspect. This is a pretty complex topic. GGML seems to use 16bit as per the article. If was pushing it to 8bit, I reckin I'd see no size improvement the GGML file? The article says they encode quantization versions in that file. Where are they defined?