Lossless LLM compression for efficient GPU inference via dynamic-length float

Re:Lossless LLM compression for efficient GPU inference via ... by spindump8930 on Friday April 25, 2025 @04:13PM (#43798074)

Yes, it could be stacked on quants. It might be that quantized activations already are more "dense" and so they can't be compressed as much (from 16 -> ~11 bits), but certainly possible.

[ Parent | on HN ]

Re:Lossless LLM compression for efficient GPU inference via ... by jasonjmcghee on Friday April 25, 2025 @06:15PM (#43799006)

I read it similarly - that this is a specific attribute of bfloat16, so the quants folks tend to run on local hardware don't have the same inefficiency to exploit

[ Parent | on HN ]