Re:Lossless LLM compression for efficient GPU inference via ...
by hchja on Friday April 25, 2025 @03:53PM
(#43797905)
This is pretty useless in any case that doesn’t involve BFloat16 models
[ Parent
| on HN ]
Re:Lossless LLM compression for efficient GPU inference via ...
by spindump8930 on Friday April 25, 2025 @04:12PM
(#43798063)
bf16 is the defacto default datatype and distribution type for LLMs, which are then often eagerly quantized by users with more limited hardware. See the recent Llama releases and e.g. the H100 spec sheet (advertised flops and metrics target bf16).
Re:Lossless LLM compression for efficient GPU inference via ...
by throwaway314155 on Friday April 25, 2025 @04:08PM
(#43798024)
So an increasingly smaller number of cases?