News for hackers. Stuff that matters.
Re:Do not buy RAM, compress by vlovich123 on Friday November 20, 2026 @01:16AM (#50570821)
Just to nitpick, it’s W_V * W_O and the “identity” matrix is not a literal identity matrix (not square and not strictly 1s and 0s) so it’s still generally not compressible.

But yes, there’s weight pruning techniques for W_O and W_V, LoRA to reduce the size of weights through approximations, and MQA, GQA, MLA to remove redundancies in the KV projection weights (and also shrink the size of the kv cache while we’re at it).

There are no comments on this comment yet.