Re:Do not buy RAM, compress
by vlovich123 on Friday November 20, 2026 @01:16AM
(#50570821)
Just to nitpick, it’s W_V * W_O and the “identity” matrix is not a literal identity matrix (not square and not strictly 1s and 0s) so it’s still generally not compressible.
But yes, there’s weight pruning techniques for W_O and W_V, LoRA to reduce the size of weights through approximations, and MQA, GQA, MLA to remove redundancies in the KV projection weights (and also shrink the size of the kv cache while we’re at it).
Consider the induction heads, their W_O are generally expected to be the identity matrix and it is useful to have many of them at many layers (e.g. if you only had one induction head in the whole network you'd only be able to perform one induction circuit per token emitted)
That being said the attention heads are a small portion of the total network size