News for hackers. Stuff that matters.
Re:Do not buy RAM, compress by Lio on Thursday November 19, 2026 @05:54AM (#50558511)
RAM compression sounds very useful but the thing I really need extra RAM for is local inference.

Since the models are effectively highly compressed data already my assumption is that RAM compression won't help improve that much (I'm assuming unified memory here as per the Framework Desktop).

Could anyone comment on that?

Re:Do not buy RAM, compress by rft on Thursday November 19, 2026 @06:11AM (#50558617)
I haven't tried it in RAM, but your reasoning matches what I saw when I looked at compressing weights for archiving them. The weights don't compress nicely with standard, especially fast, algorithms. Some more background in this repo [1] including example compression ratios. I would also worry about decompression speed, that gives you a hard bound on the throughput. If the numbers in this repo are accurate, you likely hit the throughput limit of decompression way before you hit the memory limit, even considering the space savings and thus higher effective throughput.

[1] https://github.com/zipnn/zipnn#results

Re:Do not buy RAM, compress by kalcode on Thursday November 19, 2026 @06:34AM (#50558817)
I don't know at all. But I've been playing with oMlx and Splash. Using Pi there are compression extensions like the billion-context. I suspect your right.

But all seem to have massive issue boards. What I did see was oMlx release clustering so you can string 3 macs and have 3 times the memory or use one for prefilling accelerating.

Which I thought was neat and what I'll likely poke at instead of getting more ran.

Re:Do not buy RAM, compress by 0123456789ABCDE on Thursday November 19, 2026 @07:23AM (#50559184)
[delayed]
Re:Do not buy RAM, compress by kalcode on Thursday November 19, 2026 @10:49AM (#50561671)
Yah I think I am conflating it. I was thinking using the ram more effectively to begin to squeeze more, one of those is context size. Less VRAM the less context you can handle, so there always a lot of tuning to find that sweet spot.

So my mind on that.

Re:Do not buy RAM, compress by matja on Thursday November 19, 2026 @07:45AM (#50559382)
> A bit like trying to compress a JPEG in a zip file

Interesting comparison, I wonder if there are loss-less compression schemes that are "weight-aware", about the statistical properties of model weights.

Back to that line - because (Huffman coded) JPEG images can be lossless recoded using arithmetic coding, which makes them smaller than if they were compressed with the best general-purpose compression algorithms/tools (e.g. `jpegtran -arithmetic < input.jpg > output.jpg` beats `xz -9 < input.jpg > output.jpg`). AC takes advantage of the properties of the image coding better than general-purpose compression tools on the output of the Huffman coding.

Re:Do not buy RAM, compress by rft on Thursday November 19, 2026 @08:44AM (#50560021)
The library I use for archiving compressed raw LLM weights ([1] , posted in my other comment) afaik at least reorders the weights to more easily compress them. They also have a paper linked, which I admittedly didn't read, that might have more background and tradeoffs. Likely they will have discussed methods to take advantage of the structure of weights.

[1] https://github.com/zipnn/zipnn#results

Re:Do not buy RAM, compress by strobe on Thursday November 19, 2026 @10:45AM (#50561617)
it interesting because usually on workstation the browser & electron apps it's that holding much of the memory hostage (it's very common to see that browser consuming 20-30gb on 64gb system), so may be if it will be significant compression win on that side it leads to more memory for model but how much performance penalty would be for inference in that case?
Re:Do not buy RAM, compress by throw-qqqqq on Thursday November 19, 2026 @10:55AM (#50561750)
The max theoretical ratio of (byte-independent) compression can be approximated using the Shannon entropy.

Most quantized GGUFs have a very high Shannon entropy, so are almost perfectly compressed already. A Q4_K_M quantization is often around +7.90 bits/byte and so is virtually incompressible.

Unquantized models (F16) are often around +7bits/byte for comparison.

https://en.wikipedia.org/wiki/Shannon%27s_source_coding_theo...

Re:Do not buy RAM, compress by vlovich123 on Thursday November 19, 2026 @11:13AM (#50561996)
You are correct, compression doesn’t help a model. That’s why quantization is inherently a lossy operation although there’s tricks to limit the perceived loss. That’s the only way to further compress the data.

One way to think about it is that if the weights were compressible, it’s a clear signal that better training could make the model smarter as there’s redundant information in the structure that could be used to store other meaning.

Re:Do not buy RAM, compress by Powdering7082 on Thursday November 19, 2026 @12:17PM (#50562945)
> redundant information in the structure that could be used to store other meaning.

Consider the induction heads, their W_O are generally expected to be the identity matrix and it is useful to have many of them at many layers (e.g. if you only had one induction head in the whole network you'd only be able to perform one induction circuit per token emitted)

That being said the attention heads are a small portion of the total network size

Re:Do not buy RAM, compress by vlovich123 on Friday November 20, 2026 @01:16AM (#50570821)
Just to nitpick, it’s W_V * W_O and the “identity” matrix is not a literal identity matrix (not square and not strictly 1s and 0s) so it’s still generally not compressible.

But yes, there’s weight pruning techniques for W_O and W_V, LoRA to reduce the size of weights through approximations, and MQA, GQA, MLA to remove redundancies in the KV projection weights (and also shrink the size of the kv cache while we’re at it).

Re:Do not buy RAM, compress by hobo123 on Thursday November 19, 2026 @03:03PM (#50565519)
Since you need to apply CPU time to compress memory, and also CPU before you can map pages back into process memory (by uncompressing), this is only useful for not frequently used memory, like swap only a bit faster.

I'm not too familiar with LLM memory access patterns, but since the model files are stored on disk, you wouldn't ever keep them in memory anyway (except caching) and just reread them on demand.

Re:Do not buy RAM, compress by RossBencina on Thursday November 19, 2026 @03:33PM (#50565965)
[delayed]
Re:Do not buy RAM, compress by origini on Thursday November 19, 2026 @04:08PM (#50566388)
Compression and decompression of memory is potentially even beneficial for CPU as the memory bandwidth to the CPU is much slower than the CPU itself.
Re:Do not buy RAM, compress by acchow on Thursday November 19, 2026 @06:04PM (#50567817)
I'm learning about this now. Apparently it's not as simple as comparing decompression speeds vs memory bandwidth.

Compression happens in 16KB pages. But memory reads can be 64-byte or 128-byte lines. So from a compressed page, the CPU cannot directly read a line from memory.

So to read even a single 8-byte line, the kernel will page fault, allocate a fresh 16KB page, decompress the entire 16KB page into it, update the page tables, flush the TLB, then resume execution.