News for hackers. Stuff that matters.

A comment on Lossless LLM compression for efficient GPU inference via dynamic-length float.

Re:Lossless LLM compression for efficient GPU inference via ... by Animats on Friday April 25, 2025 @05:35PM (#43798770)
Once this weight format war settles down, hardware can be built to support it. Presumably you want matrix multiply hardware optimized for whatever weight format turns out to be reasonably optimal.
[ Parent | on HN ]
Re:Lossless LLM compression for efficient GPU inference via ... by eoerl on Friday April 25, 2025 @06:53PM (#43799248)
Optimization is post hoc here : you have to train first to be able to huffman en ode, so it's not a pure format question
[ Parent | on HN ]