News for hackers. Stuff that matters.
Re:Do not buy RAM, compress by vlovich123 on Thursday November 19, 2026 @11:13AM (#50561996)
You are correct, compression doesn’t help a model. That’s why quantization is inherently a lossy operation although there’s tricks to limit the perceived loss. That’s the only way to further compress the data.

One way to think about it is that if the weights were compressible, it’s a clear signal that better training could make the model smarter as there’s redundant information in the structure that could be used to store other meaning.

Re:Do not buy RAM, compress by Powdering7082 on Thursday November 19, 2026 @12:17PM (#50562945)
> redundant information in the structure that could be used to store other meaning.

Consider the induction heads, their W_O are generally expected to be the identity matrix and it is useful to have many of them at many layers (e.g. if you only had one induction head in the whole network you'd only be able to perform one induction circuit per token emitted)

That being said the attention heads are a small portion of the total network size

Re:Do not buy RAM, compress by vlovich123 on Friday November 20, 2026 @01:16AM (#50570821)
Just to nitpick, it’s W_V * W_O and the “identity” matrix is not a literal identity matrix (not square and not strictly 1s and 0s) so it’s still generally not compressible.

But yes, there’s weight pruning techniques for W_O and W_V, LoRA to reduce the size of weights through approximations, and MQA, GQA, MLA to remove redundancies in the KV projection weights (and also shrink the size of the kv cache while we’re at it).